Started by upstream project "elastic-elasticsearch-master" build number 2760 originally caused by: Started by timer [EnvInject] - Loading node environment variables. [EnvInject] - Preparing an environment for the build. [EnvInject] - Keeping Jenkins system variables. [EnvInject] - Keeping Jenkins build variables. [EnvInject] - Injecting as environment variables the properties content HOME=C:\Users\jenkins [EnvInject] - Variables injected successfully. [EnvInject] - Injecting contributions. Building remotely on SLAVE-0633DD87F (virtual swarm win32nt windows windows-2016) in workspace C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility Wiping out workspace first. Cloning the remote Git repository Cloning repository https://github.com/elastic/elasticsearch.git > git init C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility # timeout=10 ERROR: Reference path does not exist: /var/lib/jenkins/.git-references/elasticsearch.git Fetching upstream changes from https://github.com/elastic/elasticsearch.git > git --version # timeout=10 using GIT_SSH to set credentials master node jenkins user ~/.ssh > git fetch --tags --progress https://github.com/elastic/elasticsearch.git +refs/heads/*:refs/remotes/origin/* > git config remote.origin.url https://github.com/elastic/elasticsearch.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/elastic/elasticsearch.git # timeout=10 Fetching upstream changes from https://github.com/elastic/elasticsearch.git using GIT_SSH to set credentials master node jenkins user ~/.ssh > git fetch --tags --progress https://github.com/elastic/elasticsearch.git +refs/heads/*:refs/remotes/origin/* Checking out Revision d3ee35ef18d110d36bb52bcab261035902d4ce21 (refs/remotes/origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f d3ee35ef18d110d36bb52bcab261035902d4ce21 Commit message: "[Docs] Add snippets for POS stop tags default value" > git rev-list --no-walk 10fcf30ce1c09f700ab44f46616e85ede70da17b # timeout=10 [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path '.ci/java-versions.properties' [EnvInject] - Variables injected successfully. [EnvInject] - Injecting as environment variables the properties content JAVA10_HOME=C:\Users\jenkins\.java\java10 JAVA9_HOME=C:\Users\jenkins\.java\java9 JAVA8_HOME=C:\Users\jenkins\.java\java8 JAVA7_HOME=C:\Users\jenkins\.java\java7 GRADLE_JAVA_HOME=C:\Users\jenkins\.java\$ES_RUNTIME_JAVA RUNTIME_JAVA_HOME=C:\Users\jenkins\.java\$ES_RUNTIME_JAVA JAVA_HOME=C:\Users\jenkins\.java\$ES_BUILD_JAVA JAVA11_HOME=C:\Users\jenkins\.java\java11 [EnvInject] - Variables injected successfully. [elastic+elasticsearch+master+multijob-windows-compatibility] $ cmd /c call C:\Users\JENKIN~1.SLA\AppData\Local\Temp\jenkins6302557982983030371.bat C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility>( echo call gradlew.bat --stacktrace -Dorg.gradle.java.home=C:\Users\jenkins\.java\java8 clean echo call gradlew.bat --info -Dorg.gradle.java.home=C:\Users\jenkins\.java\java8 build ) | java -jar "C:\Program Files\infra\bin\runbld" --last-good-commit elastic+elasticsearch+master+intake - runbld>>> runbld started runbld>>> 1.5.15/e747c01b37eedd5e88c6529d7604186f0826975c runbld>>> Debug logging enabled. runbld>>> BUILD: https://5086a1f436ee16623a447bdf25881bbc.us-east-1.aws.found.io:9243/build-1498188341992/t/20180505081907-74967D11 runbld>>> Adding vcs info for the latest commit: d3ee35ef18d110d36bb52bcab261035902d4ce21 runbld>>> Using last successful commit 10fcf30 from the job elastic+elasticsearch+master+intake with the build id 20180505022008-F3A832AD that was started at 2018-05-05T02:20:09.565Z and finished 2h 42m 48s ago runbld>>> BUILD_METADATA found in Elasticsearch. runbld>>> BUILD_METADATA: bwc_refspec_distribution:bwc:next-minor-snapshot=12250a1c10bb7cca742366d938b9a624e94610a2;bwc_refspec_distribution:bwc:staged-minor-snapshot=e33370a87346275a3e8989296c5f5c48676352b1;bwc_refspec_distribution:bwc:next-bugfix-snapshot=489f39af58216bf44c5e6aa944529ef31596118e runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>> runbld>>> Adding C:\Users\jenkins\.java\java10\bin to the path. C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility>call gradlew.bat --stacktrace -Dorg.gradle.java.home=C:\Users\jenkins\.java\java8 clean To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/4.7/userguide/gradle_daemon.html. Daemon will be stopped at the end of the build stopping after processing > Task :buildSrc:compileJava > Task :buildSrc:compileGroovy > Task :buildSrc:writeVersionProperties > Task :buildSrc:processResources > Task :buildSrc:classes > Task :buildSrc:jar > Task :buildSrc:assemble > Task :buildSrc:compileTestJava > Task :buildSrc:compileTestGroovy > Task :buildSrc:processTestResources NO-SOURCE > Task :buildSrc:testClasses > Task :buildSrc:test > Task :buildSrc:check > Task :buildSrc:build > Configure project :benchmarks ======================================= Elasticsearch Build Hamster says Hello! ======================================= Gradle Version : 4.7 OS Info : Windows Server 2016 10.0 (amd64) JDK Version (gradle) : Oracle Corporation 1.8.0_162 [Java HotSpot(TM) 64-Bit Server VM 25.162-b12] JAVA_HOME (gradle) : C:\Users\jenkins\.java\java8 JDK Version (compile) : Oracle Corporation 10 [Java HotSpot(TM) 64-Bit Server VM 10+46] JAVA_HOME (compile) : C:\Users\jenkins\.java\java10 JDK Version (runtime) : Oracle Corporation 1.8.0_162 [Java HotSpot(TM) 64-Bit Server VM 25.162-b12] JAVA_HOME (runtime) : C:\Users\jenkins\.java\java8 Random Testing Seed : C08856BE49616116 > Configure project :modules:reindex Disabling reindex-from-old tests because we can't get the pid file on windows > Configure project :plugins:repository-hdfs hdfsFixture unsupported, please set HADOOP_HOME and put HADOOP_HOME\bin in PATH > Task :benchmarks:clean > Task :build-tools:clean > Task :distribution:clean UP-TO-DATE > Task :docs:clean > Task :rest-api-spec:clean > Task :server:clean > Task :client:benchmark:clean > Task :client:client-benchmark-noop-api-plugin:clean > Task :client:rest:clean > Task :client:rest-high-level:clean > Task :client:sniffer:clean > Task :client:test:clean > Task :client:transport:clean > Task :distribution:archives:clean UP-TO-DATE > Task :distribution:packages:clean UP-TO-DATE > Task :example-plugins:custom-settings:clean > Task :example-plugins:meta-plugin:clean > Task :example-plugins:painless-whitelist:clean > Task :example-plugins:rescore:clean > Task :example-plugins:rest-handler:clean > Task :example-plugins:script-expert-scoring:clean > Task :libs:elasticsearch-core:clean > Task :libs:elasticsearch-nio:clean > Task :libs:grok:clean > Task :libs:plugin-classloader:clean > Task :libs:secure-sm:clean > Task :libs:x-content:clean > Task :modules:aggs-matrix-stats:clean > Task :modules:analysis-common:clean > Task :modules:ingest-common:clean > Task :modules:lang-expression:clean > Task :modules:lang-mustache:clean > Task :modules:lang-painless:clean > Task :modules:mapper-extras:clean > Task :modules:parent-join:clean > Task :modules:percolator:clean > Task :modules:rank-eval:clean > Task :modules:reindex:clean > Task :modules:repository-url:clean > Task :modules:transport-netty4:clean > Task :plugins:analysis-icu:clean > Task :plugins:analysis-kuromoji:clean > Task :plugins:analysis-nori:clean > Task :plugins:analysis-phonetic:clean > Task :plugins:analysis-smartcn:clean > Task :plugins:analysis-stempel:clean > Task :plugins:analysis-ukrainian:clean > Task :plugins:discovery-azure-classic:clean > Task :plugins:discovery-ec2:clean > Task :plugins:discovery-file:clean > Task :plugins:discovery-gce:clean > Task :plugins:ingest-attachment:clean > Task :plugins:ingest-geoip:clean > Task :plugins:ingest-user-agent:clean > Task :plugins:mapper-murmur3:clean > Task :plugins:mapper-size:clean > Task :plugins:repository-azure:clean > Task :plugins:repository-gcs:clean > Task :plugins:repository-hdfs:clean > Task :plugins:repository-s3:clean > Task :plugins:store-smb:clean > Task :plugins:transport-nio:clean > Task :qa:ccs-unavailable-clusters:clean > Task :qa:die-with-dignity:clean > Task :qa:evil-tests:clean > Task :qa:full-cluster-restart:clean > Task :qa:mixed-cluster:clean > Task :qa:multi-cluster-search:clean > Task :qa:no-bootstrap-tests:clean > Task :qa:query-builder-bwc:clean > Task :qa:rolling-upgrade:clean > Task :qa:smoke-test-client:clean > Task :qa:smoke-test-http:clean > Task :qa:smoke-test-ingest-disabled:clean > Task :qa:smoke-test-ingest-with-all-dependencies:clean > Task :qa:smoke-test-multinode:clean > Task :qa:smoke-test-plugins:clean > Task :qa:smoke-test-rank-eval-with-mustache:clean > Task :qa:smoke-test-reindex-with-all-modules:clean > Task :qa:vagrant:clean > Task :qa:verify-version-constants:clean > Task :qa:wildfly:clean > Task :server:cli:clean > Task :test:framework:clean > Task :test:logger-usage:clean > Task :x-pack:docs:clean > Task :x-pack:license-tools:clean > Task :x-pack:plugin:clean > Task :x-pack:transport-client:clean > Task :distribution:archives:integ-test-zip:clean > Task :distribution:archives:oss-tar:clean UP-TO-DATE > Task :distribution:archives:oss-zip:clean UP-TO-DATE > Task :distribution:archives:tar:clean UP-TO-DATE > Task :distribution:archives:zip:clean UP-TO-DATE > Task :distribution:bwc:maintenance-bugfix-snapshot:clean UP-TO-DATE > Task :distribution:bwc:next-bugfix-snapshot:clean UP-TO-DATE > Task :distribution:bwc:next-minor-snapshot:clean UP-TO-DATE > Task :distribution:bwc:staged-minor-snapshot:clean UP-TO-DATE > Task :distribution:packages:deb:clean UP-TO-DATE > Task :distribution:packages:oss-deb:clean UP-TO-DATE > Task :distribution:packages:oss-rpm:clean UP-TO-DATE > Task :distribution:packages:rpm:clean UP-TO-DATE > Task :distribution:tools:launchers:clean > Task :distribution:tools:plugin-cli:clean > Task :example-plugins:meta-plugin:dummy-plugin1:clean > Task :example-plugins:meta-plugin:dummy-plugin2:clean > Task :modules:lang-painless:spi:clean > Task :test:fixtures:example-fixture:clean > Task :test:fixtures:hdfs-fixture:clean > Task :test:fixtures:krb5kdc-fixture:clean > Task :test:fixtures:old-elasticsearch:clean > Task :x-pack:plugin:core:clean > Task :x-pack:plugin:deprecation:clean > Task :x-pack:plugin:graph:clean > Task :x-pack:plugin:logstash:clean > Task :x-pack:plugin:ml:clean > Task :x-pack:plugin:monitoring:clean > Task :x-pack:plugin:rollup:clean > Task :x-pack:plugin:security:clean > Task :x-pack:plugin:sql:clean > Task :x-pack:plugin:upgrade:clean > Task :x-pack:plugin:watcher:clean > Task :x-pack:qa:audit-tests:clean > Task :x-pack:qa:core-rest-tests-with-security:clean > Task :x-pack:qa:full-cluster-restart:clean > Task :x-pack:qa:ml-basic-multi-node:clean > Task :x-pack:qa:ml-disabled:clean > Task :x-pack:qa:ml-native-tests:clean > Task :x-pack:qa:ml-no-bootstrap-tests:clean > Task :x-pack:qa:ml-single-node-tests:clean > Task :x-pack:qa:multi-cluster-search-security:clean > Task :x-pack:qa:multi-node:clean > Task :x-pack:qa:openldap-tests:clean > Task :x-pack:qa:reindex-tests-with-security:clean > Task :x-pack:qa:rolling-upgrade:clean > Task :x-pack:qa:rolling-upgrade-basic:clean > Task :x-pack:qa:saml-idp-tests:clean > Task :x-pack:qa:security-client-tests:clean > Task :x-pack:qa:security-example-spi-extension:clean > Task :x-pack:qa:security-migrate-tests:clean > Task :x-pack:qa:security-setup-password-tests:clean > Task :x-pack:qa:security-tools-tests:clean > Task :x-pack:qa:smoke-test-graph-with-security:clean > Task :x-pack:qa:smoke-test-ml-with-security:clean > Task :x-pack:qa:smoke-test-monitoring-with-watcher:clean > Task :x-pack:qa:smoke-test-plugins:clean > Task :x-pack:qa:smoke-test-plugins-ssl:clean > Task :x-pack:qa:smoke-test-security-with-mustache:clean > Task :x-pack:qa:smoke-test-watcher:clean > Task :x-pack:qa:smoke-test-watcher-with-security:clean > Task :x-pack:qa:sql:clean > Task :x-pack:qa:transport-client-tests:clean > Task :x-pack:qa:vagrant:clean UP-TO-DATE > Task :x-pack:test:idp-fixture:clean > Task :x-pack:test:smb-fixture:clean > Task :plugins:repository-azure:qa:microsoft-azure-storage:clean > Task :plugins:repository-gcs:qa:google-cloud-storage:clean > Task :plugins:repository-s3:qa:amazon-s3:clean > Task :x-pack:plugin:bwc:maintenance-bugfix-snapshot:clean UP-TO-DATE > Task :x-pack:plugin:bwc:next-bugfix-snapshot:clean UP-TO-DATE > Task :x-pack:plugin:bwc:next-minor-snapshot:clean UP-TO-DATE > Task :x-pack:plugin:bwc:staged-minor-snapshot:clean UP-TO-DATE > Task :x-pack:plugin:ml:cpp-snapshot:clean UP-TO-DATE > Task :x-pack:plugin:sql:jdbc:clean > Task :x-pack:plugin:sql:sql-cli:clean > Task :x-pack:plugin:sql:sql-proto:clean > Task :x-pack:plugin:sql:sql-shared-client:clean > Task :x-pack:qa:full-cluster-restart:with-system-key:clean > Task :x-pack:qa:full-cluster-restart:without-system-key:clean > Task :x-pack:qa:rolling-upgrade:with-system-key:clean > Task :x-pack:qa:rolling-upgrade:without-system-key:clean > Task :x-pack:qa:sql:multinode:clean > Task :x-pack:qa:sql:no-security:clean > Task :x-pack:qa:sql:security:clean UP-TO-DATE > Task :x-pack:qa:third-party:active-directory:clean > Task :x-pack:qa:third-party:hipchat:clean > Task :x-pack:qa:third-party:jira:clean > Task :x-pack:qa:third-party:pagerduty:clean > Task :x-pack:qa:third-party:slack:clean > Task :x-pack:qa:sql:security:no-ssl:clean > Task :x-pack:qa:sql:security:ssl:clean Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. See https://docs.gradle.org/4.7/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 1m 4s 181 actionable tasks: 159 executed, 22 up-to-date Initialized native services in: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\native Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility Command: C:\Users\jenkins\.java\java8\bin\java.exe -version Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java.exe'' To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/4.7/userguide/gradle_daemon.html. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility Command: C:\Users\jenkins\.java\java8\bin\java.exe -version Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java.exe'' Starting process 'Gradle build daemon'. Working directory: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\daemon\4.7 Command: C:\Users\jenkins\.java\java8\bin\java.exe -Xmx1536m -Dfile.encoding=windows-1252 -Duser.country=US -Duser.language=en -Duser.variant -cp C:\Users\jenkins.SLAVE-0633DD87F\.gradle\wrapper\dists\gradle-4.7-all\4cret0dgl5o3b21weaoncl7ys\gradle-4.7\lib\gradle-launcher-4.7.jar org.gradle.launcher.daemon.bootstrap.GradleDaemon 4.7 Successfully started process 'Gradle build daemon' An attempt to start the daemon took 1.129 secs. The client will now receive all logging from the daemon (pid: 11300). The daemon log file: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\daemon\4.7\daemon-11300.out.log Daemon will be stopped at the end of the build stopping after processing Using 4 worker leases. Starting Build > Configure project :buildSrc Evaluating project ':buildSrc' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build.gradle'. Selected primary task 'build' from project : :compileJava (Thread[Daemon worker,5,main]) started. > Task :buildSrc:compileJava UP-TO-DATE Skipping task ':buildSrc:compileJava' as it is up-to-date. :compileJava (Thread[Daemon worker,5,main]) completed. Took 0.25 secs. :compileGroovy (Thread[Daemon worker,5,main]) started. > Task :buildSrc:compileGroovy UP-TO-DATE Skipping task ':buildSrc:compileGroovy' as it is up-to-date. :compileGroovy (Thread[Daemon worker,5,main]) completed. Took 0.148 secs. :writeVersionProperties (Thread[Daemon worker,5,main]) started. > Task :buildSrc:writeVersionProperties UP-TO-DATE Skipping task ':buildSrc:writeVersionProperties' as it is up-to-date. :writeVersionProperties (Thread[Daemon worker,5,main]) completed. Took 0.003 secs. :processResources (Thread[Daemon worker,5,main]) started. > Task :buildSrc:processResources UP-TO-DATE Skipping task ':buildSrc:processResources' as it is up-to-date. :processResources (Thread[Daemon worker,5,main]) completed. Took 0.036 secs. :classes (Thread[Daemon worker,5,main]) started. > Task :buildSrc:classes UP-TO-DATE Skipping task ':buildSrc:classes' as it has no actions. :classes (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :jar (Thread[Daemon worker,5,main]) started. > Task :buildSrc:jar UP-TO-DATE Skipping task ':buildSrc:jar' as it is up-to-date. :jar (Thread[Daemon worker,5,main]) completed. Took 0.015 secs. :assemble (Thread[Daemon worker,5,main]) started. > Task :buildSrc:assemble UP-TO-DATE Skipping task ':buildSrc:assemble' as it has no actions. :assemble (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :compileTestJava (Thread[Daemon worker,5,main]) started. > Task :buildSrc:compileTestJava UP-TO-DATE Skipping task ':buildSrc:compileTestJava' as it is up-to-date. :compileTestJava (Thread[Daemon worker,5,main]) completed. Took 0.11 secs. :compileTestGroovy (Thread[Daemon worker,5,main]) started. > Task :buildSrc:compileTestGroovy UP-TO-DATE Skipping task ':buildSrc:compileTestGroovy' as it is up-to-date. :compileTestGroovy (Thread[Daemon worker,5,main]) completed. Took 0.067 secs. :processTestResources (Thread[Daemon worker,5,main]) started. > Task :buildSrc:processTestResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\src\test\resources', not found Skipping task ':buildSrc:processTestResources' as it has no source files and no previous output files. :processTestResources (Thread[Daemon worker,5,main]) completed. Took 0.003 secs. :testClasses (Thread[Daemon worker,5,main]) started. > Task :buildSrc:testClasses UP-TO-DATE Skipping task ':buildSrc:testClasses' as it has no actions. :testClasses (Thread[Daemon worker,5,main]) completed. Took 0.002 secs. :test (Thread[Daemon worker,5,main]) started. > Task :buildSrc:test UP-TO-DATE Skipping task ':buildSrc:test' as it is up-to-date. :test (Thread[Daemon worker,5,main]) completed. Took 0.072 secs. :check (Thread[Daemon worker,5,main]) started. > Task :buildSrc:check UP-TO-DATE Skipping task ':buildSrc:check' as it has no actions. :check (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :build (Thread[Daemon worker,5,main]) started. > Task :buildSrc:build UP-TO-DATE Skipping task ':buildSrc:build' as it has no actions. :build (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. Settings evaluated using settings file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\settings.gradle'. Projects loaded. Root project using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\build.gradle'. Included projects: [root project 'elastic+elasticsearch+master+multijob-windows-compatibility', project ':benchmarks', project ':build-tools', project ':client', project ':distribution', project ':docs', project ':example-plugins', project ':libs', project ':modules', project ':plugins', project ':qa', project ':rest-api-spec', project ':server', project ':test', project ':x-pack', project ':client:benchmark', project ':client:client-benchmark-noop-api-plugin', project ':client:rest', project ':client:rest-high-level', project ':client:sniffer', project ':client:test', project ':client:transport', project ':distribution:archives', project ':distribution:bwc', project ':distribution:packages', project ':distribution:tools', project ':example-plugins:custom-settings', project ':example-plugins:meta-plugin', project ':example-plugins:painless-whitelist', project ':example-plugins:rescore', project ':example-plugins:rest-handler', project ':example-plugins:script-expert-scoring', project ':libs:elasticsearch-core', project ':libs:elasticsearch-nio', project ':libs:grok', project ':libs:plugin-classloader', project ':libs:secure-sm', project ':libs:x-content', project ':modules:aggs-matrix-stats', project ':modules:analysis-common', project ':modules:ingest-common', project ':modules:lang-expression', project ':modules:lang-mustache', project ':modules:lang-painless', project ':modules:mapper-extras', project ':modules:parent-join', project ':modules:percolator', project ':modules:rank-eval', project ':modules:reindex', project ':modules:repository-url', project ':modules:transport-netty4', project ':plugins:analysis-icu', project ':plugins:analysis-kuromoji', project ':plugins:analysis-nori', project ':plugins:analysis-phonetic', project ':plugins:analysis-smartcn', project ':plugins:analysis-stempel', project ':plugins:analysis-ukrainian', project ':plugins:discovery-azure-classic', project ':plugins:discovery-ec2', project ':plugins:discovery-file', project ':plugins:discovery-gce', project ':plugins:ingest-attachment', project ':plugins:ingest-geoip', project ':plugins:ingest-user-agent', project ':plugins:mapper-murmur3', project ':plugins:mapper-size', project ':plugins:repository-azure', project ':plugins:repository-gcs', project ':plugins:repository-hdfs', project ':plugins:repository-s3', project ':plugins:store-smb', project ':plugins:transport-nio', project ':qa:ccs-unavailable-clusters', project ':qa:die-with-dignity', project ':qa:evil-tests', project ':qa:full-cluster-restart', project ':qa:mixed-cluster', project ':qa:multi-cluster-search', project ':qa:no-bootstrap-tests', project ':qa:query-builder-bwc', project ':qa:rolling-upgrade', project ':qa:smoke-test-client', project ':qa:smoke-test-http', project ':qa:smoke-test-ingest-disabled', project ':qa:smoke-test-ingest-with-all-dependencies', project ':qa:smoke-test-multinode', project ':qa:smoke-test-plugins', project ':qa:smoke-test-rank-eval-with-mustache', project ':qa:smoke-test-reindex-with-all-modules', project ':qa:vagrant', project ':qa:verify-version-constants', project ':qa:wildfly', project ':server:cli', project ':test:fixtures', project ':test:framework', project ':test:logger-usage', project ':x-pack:docs', project ':x-pack:license-tools', project ':x-pack:plugin', project ':x-pack:qa', project ':x-pack:test', project ':x-pack:transport-client', project ':distribution:archives:integ-test-zip', project ':distribution:archives:oss-tar', project ':distribution:archives:oss-zip', project ':distribution:archives:tar', project ':distribution:archives:zip', project ':distribution:bwc:maintenance-bugfix-snapshot', project ':distribution:bwc:next-bugfix-snapshot', project ':distribution:bwc:next-minor-snapshot', project ':distribution:bwc:staged-minor-snapshot', project ':distribution:packages:deb', project ':distribution:packages:oss-deb', project ':distribution:packages:oss-rpm', project ':distribution:packages:rpm', project ':distribution:tools:launchers', project ':distribution:tools:plugin-cli', project ':example-plugins:meta-plugin:dummy-plugin1', project ':example-plugins:meta-plugin:dummy-plugin2', project ':modules:lang-painless:spi', project ':plugins:repository-azure:qa', project ':plugins:repository-gcs:qa', project ':plugins:repository-s3:qa', project ':test:fixtures:example-fixture', project ':test:fixtures:hdfs-fixture', project ':test:fixtures:krb5kdc-fixture', project ':test:fixtures:old-elasticsearch', project ':x-pack:plugin:bwc', project ':x-pack:plugin:core', project ':x-pack:plugin:deprecation', project ':x-pack:plugin:graph', project ':x-pack:plugin:logstash', project ':x-pack:plugin:ml', project ':x-pack:plugin:monitoring', project ':x-pack:plugin:rollup', project ':x-pack:plugin:security', project ':x-pack:plugin:sql', project ':x-pack:plugin:upgrade', project ':x-pack:plugin:watcher', project ':x-pack:qa:audit-tests', project ':x-pack:qa:core-rest-tests-with-security', project ':x-pack:qa:full-cluster-restart', project ':x-pack:qa:ml-basic-multi-node', project ':x-pack:qa:ml-disabled', project ':x-pack:qa:ml-native-tests', project ':x-pack:qa:ml-no-bootstrap-tests', project ':x-pack:qa:ml-single-node-tests', project ':x-pack:qa:multi-cluster-search-security', project ':x-pack:qa:multi-node', project ':x-pack:qa:openldap-tests', project ':x-pack:qa:reindex-tests-with-security', project ':x-pack:qa:rolling-upgrade', project ':x-pack:qa:rolling-upgrade-basic', project ':x-pack:qa:saml-idp-tests', project ':x-pack:qa:security-client-tests', project ':x-pack:qa:security-example-spi-extension', project ':x-pack:qa:security-migrate-tests', project ':x-pack:qa:security-setup-password-tests', project ':x-pack:qa:security-tools-tests', project ':x-pack:qa:smoke-test-graph-with-security', project ':x-pack:qa:smoke-test-ml-with-security', project ':x-pack:qa:smoke-test-monitoring-with-watcher', project ':x-pack:qa:smoke-test-plugins', project ':x-pack:qa:smoke-test-plugins-ssl', project ':x-pack:qa:smoke-test-security-with-mustache', project ':x-pack:qa:smoke-test-watcher', project ':x-pack:qa:smoke-test-watcher-with-security', project ':x-pack:qa:sql', project ':x-pack:qa:third-party', project ':x-pack:qa:transport-client-tests', project ':x-pack:qa:vagrant', project ':x-pack:test:idp-fixture', project ':x-pack:test:smb-fixture', project ':plugins:repository-azure:qa:microsoft-azure-storage', project ':plugins:repository-gcs:qa:google-cloud-storage', project ':plugins:repository-s3:qa:amazon-s3', project ':x-pack:plugin:bwc:maintenance-bugfix-snapshot', project ':x-pack:plugin:bwc:next-bugfix-snapshot', project ':x-pack:plugin:bwc:next-minor-snapshot', project ':x-pack:plugin:bwc:staged-minor-snapshot', project ':x-pack:plugin:ml:cpp-snapshot', project ':x-pack:plugin:sql:jdbc', project ':x-pack:plugin:sql:sql-cli', project ':x-pack:plugin:sql:sql-proto', project ':x-pack:plugin:sql:sql-shared-client', project ':x-pack:qa:full-cluster-restart:with-system-key', project ':x-pack:qa:full-cluster-restart:without-system-key', project ':x-pack:qa:rolling-upgrade:with-system-key', project ':x-pack:qa:rolling-upgrade:without-system-key', project ':x-pack:qa:sql:multinode', project ':x-pack:qa:sql:no-security', project ':x-pack:qa:sql:security', project ':x-pack:qa:third-party:active-directory', project ':x-pack:qa:third-party:hipchat', project ':x-pack:qa:third-party:jira', project ':x-pack:qa:third-party:pagerduty', project ':x-pack:qa:third-party:slack', project ':x-pack:qa:sql:security:no-ssl', project ':x-pack:qa:sql:security:ssl'] > Configure project : Evaluating root project 'elastic+elasticsearch+master+multijob-windows-compatibility' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\build.gradle'. > Configure project :benchmarks Evaluating project ':benchmarks' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build.gradle'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\jrunscript''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks Command: C:\Users\jenkins\.java\java10\bin\jrunscript -e print(java.lang.System.getProperty(\"java.vendor\") + \" \" + java.lang.System.getProperty(\"java.version\") + \" [\" + java.lang.System.getProperty(\"java.vm.name\") + \" \" + java.lang.System.getProperty(\"java.vm.version\") + \"]\"); Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\jrunscript'' Starting process 'command 'C:\Users\jenkins\.java\java10\bin\jrunscript''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks Command: C:\Users\jenkins\.java\java10\bin\jrunscript -e print(java.lang.System.getProperty(\"java.specification.version\")); Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\jrunscript'' ======================================= Elasticsearch Build Hamster says Hello! ======================================= Gradle Version : 4.7 OS Info : Windows Server 2016 10.0 (amd64) JDK Version (gradle) : Oracle Corporation 1.8.0_162 [Java HotSpot(TM) 64-Bit Server VM 25.162-b12] JAVA_HOME (gradle) : C:\Users\jenkins\.java\java8 JDK Version (compile) : Oracle Corporation 10 [Java HotSpot(TM) 64-Bit Server VM 10+46] JAVA_HOME (compile) : C:\Users\jenkins\.java\java10 JDK Version (runtime) : Oracle Corporation 1.8.0_162 [Java HotSpot(TM) 64-Bit Server VM 25.162-b12] JAVA_HOME (runtime) : C:\Users\jenkins\.java\java8 Random Testing Seed : 7793AE0DA1B17AFE Starting process 'command 'C:\Users\jenkins\.java\java7\bin\jrunscript''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks Command: C:\Users\jenkins\.java\java7\bin\jrunscript -e print(java.lang.System.getProperty(\"java.specification.version\")); Successfully started process 'command 'C:\Users\jenkins\.java\java7\bin\jrunscript'' Starting process 'command 'C:\Users\jenkins\.java\java8\bin\jrunscript''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks Command: C:\Users\jenkins\.java\java8\bin\jrunscript -e print(java.lang.System.getProperty(\"java.specification.version\")); Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\jrunscript'' Starting process 'command 'C:\Users\jenkins\.java\java9\bin\jrunscript''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks Command: C:\Users\jenkins\.java\java9\bin\jrunscript -e print(java.lang.System.getProperty(\"java.specification.version\")); Successfully started process 'command 'C:\Users\jenkins\.java\java9\bin\jrunscript'' Starting process 'command 'C:\Users\jenkins\.java\java10\bin\jrunscript''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks Command: C:\Users\jenkins\.java\java10\bin\jrunscript -e print(java.lang.System.getProperty(\"java.specification.version\")); Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\jrunscript'' You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :build-tools Evaluating project ':build-tools' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :client Evaluating project ':client' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\build.gradle'. > Configure project :distribution Evaluating project ':distribution' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\build.gradle'. > Configure project :docs Evaluating project ':docs' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :example-plugins Evaluating project ':example-plugins' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\examples\build.gradle'. > Configure project :libs Evaluating project ':libs' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :modules Evaluating project ':modules' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :plugins Evaluating project ':plugins' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa Evaluating project ':qa' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\build.gradle'. > Configure project :rest-api-spec Evaluating project ':rest-api-spec' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\rest-api-spec\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :server Evaluating project ':server' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :test Evaluating project ':test' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack Evaluating project ':x-pack' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\build.gradle'. > Configure project :client:benchmark Evaluating project ':client:benchmark' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\benchmark\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :client:client-benchmark-noop-api-plugin Evaluating project ':client:client-benchmark-noop-api-plugin' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\client-benchmark-noop-api-plugin\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :client:rest Evaluating project ':client:rest' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :client:rest-high-level Evaluating project ':client:rest-high-level' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest-high-level\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :client:sniffer Evaluating project ':client:sniffer' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\sniffer\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :client:test Evaluating project ':client:test' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\test\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :client:transport Evaluating project ':client:transport' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\transport\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :distribution:archives Evaluating project ':distribution:archives' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\archives\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :distribution:bwc Evaluating project ':distribution:bwc' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\bwc\build.gradle'. > Configure project :distribution:packages Evaluating project ':distribution:packages' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\packages\build.gradle'. Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@1baf2113 Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@1baf2113 Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@1baf2113 Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@1c9511e Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@5a555f97 Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@6caf3a5c Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@242ec149 Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@6712f457 Adding user to org.gradle.api.internal.file.copy.SingleParentCopySpec@6712f457 Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@6712f457 Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@5ffcf362 Adding user to org.gradle.api.internal.file.copy.SingleParentCopySpec@5ffcf362 Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@5ffcf362 Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@37de597d Adding user to org.gradle.api.internal.file.copy.SingleParentCopySpec@37de597d Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@37de597d Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@684b3328 Adding user to org.gradle.api.internal.file.copy.SingleParentCopySpec@684b3328 Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@684b3328 Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@13dc4914 Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@13dc4914 Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@13dc4914 Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@148f28f4 Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@7c4b14ac Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@dae66d8 Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@3d372fd8 Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@50926802 Adding user to org.gradle.api.internal.file.copy.SingleParentCopySpec@50926802 Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@50926802 Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@7458f8c8 Adding user to org.gradle.api.internal.file.copy.SingleParentCopySpec@7458f8c8 Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@7458f8c8 Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@abec83d Adding user to org.gradle.api.internal.file.copy.SingleParentCopySpec@abec83d Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@abec83d Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@e736f5d Adding user to org.gradle.api.internal.file.copy.SingleParentCopySpec@e736f5d Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@e736f5d Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@6ef6e743 Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@6ef6e743 Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@6ef6e743 Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@2bbbe590 Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@1323d71a Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@4ab6f6c2 Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@6e37bcf9 Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@63212d22 Adding user to org.gradle.api.internal.file.copy.SingleParentCopySpec@63212d22 Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@63212d22 Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@1e5cd762 Adding user to org.gradle.api.internal.file.copy.SingleParentCopySpec@1e5cd762 Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@1e5cd762 Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@26da1782 Adding user to org.gradle.api.internal.file.copy.SingleParentCopySpec@26da1782 Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@26da1782 Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@75d9a7d4 Adding user to org.gradle.api.internal.file.copy.SingleParentCopySpec@75d9a7d4 Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@75d9a7d4 Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@5034353b Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@5034353b Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@5034353b Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@348101d9 Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@fd18878 Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@1b3017b6 Adding fileType to org.gradle.api.internal.file.copy.SingleParentCopySpec@68f3cf8c Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@54408482 Adding user to org.gradle.api.internal.file.copy.SingleParentCopySpec@54408482 Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@54408482 Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@57d1543a Adding user to org.gradle.api.internal.file.copy.SingleParentCopySpec@57d1543a Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@57d1543a Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@4c4774d1 Adding user to org.gradle.api.internal.file.copy.SingleParentCopySpec@4c4774d1 Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@4c4774d1 Adding createDirectoryEntry to org.gradle.api.internal.file.copy.SingleParentCopySpec@37a753d4 Adding user to org.gradle.api.internal.file.copy.SingleParentCopySpec@37a753d4 Adding permissionGroup to org.gradle.api.internal.file.copy.SingleParentCopySpec@37a753d4 > Configure project :distribution:tools Evaluating project ':distribution:tools' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\tools\build.gradle'. > Configure project :example-plugins:custom-settings Evaluating project ':example-plugins:custom-settings' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\examples\custom-settings\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :example-plugins:meta-plugin Evaluating project ':example-plugins:meta-plugin' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\examples\meta-plugin\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :example-plugins:painless-whitelist Evaluating project ':example-plugins:painless-whitelist' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\examples\painless-whitelist\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :example-plugins:rescore Evaluating project ':example-plugins:rescore' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\examples\rescore\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :example-plugins:rest-handler Evaluating project ':example-plugins:rest-handler' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\examples\rest-handler\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :example-plugins:script-expert-scoring Evaluating project ':example-plugins:script-expert-scoring' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\examples\script-expert-scoring\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :libs:elasticsearch-core Evaluating project ':libs:elasticsearch-core' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build.gradle'. > Configure project :libs:elasticsearch-nio Evaluating project ':libs:elasticsearch-nio' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build.gradle'. > Configure project :libs:grok Evaluating project ':libs:grok' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\grok\build.gradle'. > Configure project :libs:plugin-classloader Evaluating project ':libs:plugin-classloader' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader\build.gradle'. > Configure project :libs:secure-sm Evaluating project ':libs:secure-sm' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build.gradle'. > Configure project :libs:x-content Evaluating project ':libs:x-content' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build.gradle'. > Configure project :modules:aggs-matrix-stats Evaluating project ':modules:aggs-matrix-stats' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\aggs-matrix-stats\build.gradle'. > Configure project :modules:analysis-common Evaluating project ':modules:analysis-common' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\analysis-common\build.gradle'. > Configure project :modules:ingest-common Evaluating project ':modules:ingest-common' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\ingest-common\build.gradle'. > Configure project :modules:lang-expression Evaluating project ':modules:lang-expression' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\lang-expression\build.gradle'. > Configure project :modules:lang-mustache Evaluating project ':modules:lang-mustache' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\lang-mustache\build.gradle'. > Configure project :modules:lang-painless Evaluating project ':modules:lang-painless' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\lang-painless\build.gradle'. > Configure project :modules:mapper-extras Evaluating project ':modules:mapper-extras' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\mapper-extras\build.gradle'. > Configure project :modules:parent-join Evaluating project ':modules:parent-join' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\parent-join\build.gradle'. > Configure project :modules:percolator Evaluating project ':modules:percolator' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\percolator\build.gradle'. > Configure project :modules:rank-eval Evaluating project ':modules:rank-eval' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\rank-eval\build.gradle'. > Configure project :modules:reindex Evaluating project ':modules:reindex' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\reindex\build.gradle'. Disabling reindex-from-old tests because we can't get the pid file on windows > Configure project :modules:repository-url Evaluating project ':modules:repository-url' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\repository-url\build.gradle'. > Configure project :modules:transport-netty4 Evaluating project ':modules:transport-netty4' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\transport-netty4\build.gradle'. > Configure project :plugins:analysis-icu Evaluating project ':plugins:analysis-icu' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-icu\build.gradle'. > Configure project :plugins:analysis-kuromoji Evaluating project ':plugins:analysis-kuromoji' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-kuromoji\build.gradle'. > Configure project :plugins:analysis-nori Evaluating project ':plugins:analysis-nori' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-nori\build.gradle'. > Configure project :plugins:analysis-phonetic Evaluating project ':plugins:analysis-phonetic' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-phonetic\build.gradle'. > Configure project :plugins:analysis-smartcn Evaluating project ':plugins:analysis-smartcn' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-smartcn\build.gradle'. > Configure project :plugins:analysis-stempel Evaluating project ':plugins:analysis-stempel' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-stempel\build.gradle'. > Configure project :plugins:analysis-ukrainian Evaluating project ':plugins:analysis-ukrainian' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-ukrainian\build.gradle'. > Configure project :plugins:discovery-azure-classic Evaluating project ':plugins:discovery-azure-classic' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-azure-classic\build.gradle'. > Configure project :plugins:discovery-ec2 Evaluating project ':plugins:discovery-ec2' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-ec2\build.gradle'. > Configure project :plugins:discovery-file Evaluating project ':plugins:discovery-file' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-file\build.gradle'. > Configure project :plugins:discovery-gce Evaluating project ':plugins:discovery-gce' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-gce\build.gradle'. > Configure project :plugins:ingest-attachment Evaluating project ':plugins:ingest-attachment' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\ingest-attachment\build.gradle'. > Configure project :plugins:ingest-geoip Evaluating project ':plugins:ingest-geoip' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\ingest-geoip\build.gradle'. > Configure project :plugins:ingest-user-agent Evaluating project ':plugins:ingest-user-agent' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\ingest-user-agent\build.gradle'. > Configure project :plugins:mapper-murmur3 Evaluating project ':plugins:mapper-murmur3' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\mapper-murmur3\build.gradle'. > Configure project :plugins:mapper-size Evaluating project ':plugins:mapper-size' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\mapper-size\build.gradle'. > Configure project :plugins:repository-azure Evaluating project ':plugins:repository-azure' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\repository-azure\build.gradle'. > Configure project :plugins:repository-gcs Evaluating project ':plugins:repository-gcs' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\repository-gcs\build.gradle'. > Configure project :plugins:repository-hdfs Evaluating project ':plugins:repository-hdfs' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\repository-hdfs\build.gradle'. Starting process 'command 'vagrant''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\repository-hdfs Command: vagrant --version Starting process 'command 'vboxmanage''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\repository-hdfs Command: vboxmanage --version hdfsFixture unsupported, please set HADOOP_HOME and put HADOOP_HOME\bin in PATH > Configure project :plugins:repository-s3 Evaluating project ':plugins:repository-s3' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\repository-s3\build.gradle'. > Configure project :plugins:store-smb Evaluating project ':plugins:store-smb' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\store-smb\build.gradle'. > Configure project :plugins:transport-nio Evaluating project ':plugins:transport-nio' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\transport-nio\build.gradle'. > Configure project :qa:ccs-unavailable-clusters Evaluating project ':qa:ccs-unavailable-clusters' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\ccs-unavailable-clusters\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:die-with-dignity Evaluating project ':qa:die-with-dignity' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\die-with-dignity\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:evil-tests Evaluating project ':qa:evil-tests' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\evil-tests\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:full-cluster-restart Evaluating project ':qa:full-cluster-restart' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\full-cluster-restart\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:mixed-cluster Evaluating project ':qa:mixed-cluster' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\mixed-cluster\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:multi-cluster-search Evaluating project ':qa:multi-cluster-search' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\multi-cluster-search\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:no-bootstrap-tests Evaluating project ':qa:no-bootstrap-tests' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\no-bootstrap-tests\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:query-builder-bwc Evaluating project ':qa:query-builder-bwc' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\query-builder-bwc\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:rolling-upgrade Evaluating project ':qa:rolling-upgrade' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\rolling-upgrade\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:smoke-test-client Evaluating project ':qa:smoke-test-client' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\smoke-test-client\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:smoke-test-http Evaluating project ':qa:smoke-test-http' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\smoke-test-http\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:smoke-test-ingest-disabled Evaluating project ':qa:smoke-test-ingest-disabled' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\smoke-test-ingest-disabled\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:smoke-test-ingest-with-all-dependencies Evaluating project ':qa:smoke-test-ingest-with-all-dependencies' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\smoke-test-ingest-with-all-dependencies\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:smoke-test-multinode Evaluating project ':qa:smoke-test-multinode' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\smoke-test-multinode\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:smoke-test-plugins Evaluating project ':qa:smoke-test-plugins' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\smoke-test-plugins\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:smoke-test-rank-eval-with-mustache Evaluating project ':qa:smoke-test-rank-eval-with-mustache' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\smoke-test-rank-eval-with-mustache\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:smoke-test-reindex-with-all-modules Evaluating project ':qa:smoke-test-reindex-with-all-modules' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\smoke-test-reindex-with-all-modules\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:vagrant Evaluating project ':qa:vagrant' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\vagrant\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:verify-version-constants Evaluating project ':qa:verify-version-constants' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\verify-version-constants\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :qa:wildfly Evaluating project ':qa:wildfly' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\qa\wildfly\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :server:cli Evaluating project ':server:cli' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :test:fixtures Evaluating project ':test:fixtures' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\fixtures\build.gradle'. > Configure project :test:framework Evaluating project ':test:framework' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build.gradle'. > Configure project :test:logger-usage Evaluating project ':test:logger-usage' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build.gradle'. > Configure project :x-pack:docs Evaluating project ':x-pack:docs' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\docs\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:license-tools Evaluating project ':x-pack:license-tools' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\license-tools\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:plugin Evaluating project ':x-pack:plugin' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa Evaluating project ':x-pack:qa' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\build.gradle'. > Configure project :x-pack:test Evaluating project ':x-pack:test' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\test\build.gradle'. > Configure project :x-pack:transport-client Evaluating project ':x-pack:transport-client' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\transport-client\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :distribution:archives:integ-test-zip Evaluating project ':distribution:archives:integ-test-zip' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\archives\integ-test-zip\build.gradle'. > Configure project :distribution:archives:oss-tar Evaluating project ':distribution:archives:oss-tar' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\archives\oss-tar\build.gradle'. > Configure project :distribution:archives:oss-zip Evaluating project ':distribution:archives:oss-zip' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\archives\oss-zip\build.gradle'. > Configure project :distribution:archives:tar Evaluating project ':distribution:archives:tar' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\archives\tar\build.gradle'. > Configure project :distribution:archives:zip Evaluating project ':distribution:archives:zip' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\archives\zip\build.gradle'. > Configure project :distribution:bwc:maintenance-bugfix-snapshot Evaluating project ':distribution:bwc:maintenance-bugfix-snapshot' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\bwc\maintenance-bugfix-snapshot\build.gradle'. > Configure project :distribution:bwc:next-bugfix-snapshot Evaluating project ':distribution:bwc:next-bugfix-snapshot' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\bwc\next-bugfix-snapshot\build.gradle'. > Configure project :distribution:bwc:next-minor-snapshot Evaluating project ':distribution:bwc:next-minor-snapshot' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\bwc\next-minor-snapshot\build.gradle'. > Configure project :distribution:bwc:staged-minor-snapshot Evaluating project ':distribution:bwc:staged-minor-snapshot' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\bwc\staged-minor-snapshot\build.gradle'. > Configure project :distribution:packages:deb Evaluating project ':distribution:packages:deb' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\packages\deb\build.gradle'. > Configure project :distribution:packages:oss-deb Evaluating project ':distribution:packages:oss-deb' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\packages\oss-deb\build.gradle'. > Configure project :distribution:packages:oss-rpm Evaluating project ':distribution:packages:oss-rpm' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\packages\oss-rpm\build.gradle'. > Configure project :distribution:packages:rpm Evaluating project ':distribution:packages:rpm' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\packages\rpm\build.gradle'. > Configure project :distribution:tools:launchers Evaluating project ':distribution:tools:launchers' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\tools\launchers\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :distribution:tools:plugin-cli Evaluating project ':distribution:tools:plugin-cli' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\tools\plugin-cli\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :example-plugins:meta-plugin:dummy-plugin1 Evaluating project ':example-plugins:meta-plugin:dummy-plugin1' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\examples\meta-plugin\dummy-plugin1\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :example-plugins:meta-plugin:dummy-plugin2 Evaluating project ':example-plugins:meta-plugin:dummy-plugin2' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\examples\meta-plugin\dummy-plugin2\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :modules:lang-painless:spi Evaluating project ':modules:lang-painless:spi' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\lang-painless\spi\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :plugins:repository-azure:qa Evaluating project ':plugins:repository-azure:qa' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\repository-azure\qa\build.gradle'. > Configure project :plugins:repository-gcs:qa Evaluating project ':plugins:repository-gcs:qa' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\repository-gcs\qa\build.gradle'. > Configure project :plugins:repository-s3:qa Evaluating project ':plugins:repository-s3:qa' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\repository-s3\qa\build.gradle'. > Configure project :test:fixtures:example-fixture Evaluating project ':test:fixtures:example-fixture' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\fixtures\example-fixture\build.gradle'. > Configure project :test:fixtures:hdfs-fixture Evaluating project ':test:fixtures:hdfs-fixture' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\fixtures\hdfs-fixture\build.gradle'. > Configure project :test:fixtures:krb5kdc-fixture Evaluating project ':test:fixtures:krb5kdc-fixture' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\fixtures\krb5kdc-fixture\build.gradle'. > Configure project :test:fixtures:old-elasticsearch Evaluating project ':test:fixtures:old-elasticsearch' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\fixtures\old-elasticsearch\build.gradle'. > Configure project :x-pack:plugin:bwc Evaluating project ':x-pack:plugin:bwc' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\bwc\build.gradle'. > Configure project :x-pack:plugin:core Evaluating project ':x-pack:plugin:core' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\core\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:plugin:deprecation Evaluating project ':x-pack:plugin:deprecation' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\deprecation\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:plugin:graph Evaluating project ':x-pack:plugin:graph' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\graph\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:plugin:logstash Evaluating project ':x-pack:plugin:logstash' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\logstash\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:plugin:ml Evaluating project ':x-pack:plugin:ml' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\ml\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:plugin:monitoring Evaluating project ':x-pack:plugin:monitoring' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\monitoring\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:plugin:rollup Evaluating project ':x-pack:plugin:rollup' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\rollup\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:plugin:security Evaluating project ':x-pack:plugin:security' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\security\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:plugin:sql Evaluating project ':x-pack:plugin:sql' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\sql\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:plugin:upgrade Evaluating project ':x-pack:plugin:upgrade' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\upgrade\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:plugin:watcher Evaluating project ':x-pack:plugin:watcher' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\watcher\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:audit-tests Evaluating project ':x-pack:qa:audit-tests' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\audit-tests\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:core-rest-tests-with-security Evaluating project ':x-pack:qa:core-rest-tests-with-security' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\core-rest-tests-with-security\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:full-cluster-restart Evaluating project ':x-pack:qa:full-cluster-restart' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\full-cluster-restart\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:ml-basic-multi-node Evaluating project ':x-pack:qa:ml-basic-multi-node' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\ml-basic-multi-node\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:ml-disabled Evaluating project ':x-pack:qa:ml-disabled' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\ml-disabled\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:ml-native-tests Evaluating project ':x-pack:qa:ml-native-tests' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\ml-native-tests\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:ml-no-bootstrap-tests Evaluating project ':x-pack:qa:ml-no-bootstrap-tests' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\ml-no-bootstrap-tests\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:ml-single-node-tests Evaluating project ':x-pack:qa:ml-single-node-tests' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\ml-single-node-tests\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:multi-cluster-search-security Evaluating project ':x-pack:qa:multi-cluster-search-security' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\multi-cluster-search-security\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:multi-node Evaluating project ':x-pack:qa:multi-node' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\multi-node\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:test:idp-fixture Evaluating project ':x-pack:test:idp-fixture' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\test\idp-fixture\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:openldap-tests Evaluating project ':x-pack:qa:openldap-tests' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\openldap-tests\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:reindex-tests-with-security Evaluating project ':x-pack:qa:reindex-tests-with-security' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\reindex-tests-with-security\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:rolling-upgrade Evaluating project ':x-pack:qa:rolling-upgrade' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\rolling-upgrade\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:rolling-upgrade-basic Evaluating project ':x-pack:qa:rolling-upgrade-basic' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\rolling-upgrade-basic\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:saml-idp-tests Evaluating project ':x-pack:qa:saml-idp-tests' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\saml-idp-tests\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:security-client-tests Evaluating project ':x-pack:qa:security-client-tests' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\security-client-tests\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:security-example-spi-extension Evaluating project ':x-pack:qa:security-example-spi-extension' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\security-example-spi-extension\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:security-migrate-tests Evaluating project ':x-pack:qa:security-migrate-tests' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\security-migrate-tests\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:security-setup-password-tests Evaluating project ':x-pack:qa:security-setup-password-tests' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\security-setup-password-tests\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:security-tools-tests Evaluating project ':x-pack:qa:security-tools-tests' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\security-tools-tests\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:smoke-test-graph-with-security Evaluating project ':x-pack:qa:smoke-test-graph-with-security' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\smoke-test-graph-with-security\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:smoke-test-ml-with-security Evaluating project ':x-pack:qa:smoke-test-ml-with-security' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\smoke-test-ml-with-security\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:smoke-test-monitoring-with-watcher Evaluating project ':x-pack:qa:smoke-test-monitoring-with-watcher' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\smoke-test-monitoring-with-watcher\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:smoke-test-plugins Evaluating project ':x-pack:qa:smoke-test-plugins' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\smoke-test-plugins\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:smoke-test-plugins-ssl Evaluating project ':x-pack:qa:smoke-test-plugins-ssl' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\smoke-test-plugins-ssl\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:smoke-test-security-with-mustache Evaluating project ':x-pack:qa:smoke-test-security-with-mustache' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\smoke-test-security-with-mustache\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:smoke-test-watcher Evaluating project ':x-pack:qa:smoke-test-watcher' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\smoke-test-watcher\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:smoke-test-watcher-with-security Evaluating project ':x-pack:qa:smoke-test-watcher-with-security' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\smoke-test-watcher-with-security\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:sql Evaluating project ':x-pack:qa:sql' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\sql\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:third-party Evaluating project ':x-pack:qa:third-party' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\third-party\build.gradle'. > Configure project :x-pack:qa:transport-client-tests Evaluating project ':x-pack:qa:transport-client-tests' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\transport-client-tests\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:vagrant Evaluating project ':x-pack:qa:vagrant' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\vagrant\build.gradle'. > Configure project :x-pack:test:smb-fixture Evaluating project ':x-pack:test:smb-fixture' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\test\smb-fixture\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :plugins:repository-azure:qa:microsoft-azure-storage Evaluating project ':plugins:repository-azure:qa:microsoft-azure-storage' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\repository-azure\qa\microsoft-azure-storage\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :plugins:repository-gcs:qa:google-cloud-storage Evaluating project ':plugins:repository-gcs:qa:google-cloud-storage' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\repository-gcs\qa\google-cloud-storage\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :plugins:repository-s3:qa:amazon-s3 Evaluating project ':plugins:repository-s3:qa:amazon-s3' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\repository-s3\qa\amazon-s3\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:plugin:bwc:maintenance-bugfix-snapshot Evaluating project ':x-pack:plugin:bwc:maintenance-bugfix-snapshot' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\bwc\maintenance-bugfix-snapshot\build.gradle'. > Configure project :x-pack:plugin:bwc:next-bugfix-snapshot Evaluating project ':x-pack:plugin:bwc:next-bugfix-snapshot' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\bwc\next-bugfix-snapshot\build.gradle'. > Configure project :x-pack:plugin:bwc:next-minor-snapshot Evaluating project ':x-pack:plugin:bwc:next-minor-snapshot' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\bwc\next-minor-snapshot\build.gradle'. > Configure project :x-pack:plugin:bwc:staged-minor-snapshot Evaluating project ':x-pack:plugin:bwc:staged-minor-snapshot' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\bwc\staged-minor-snapshot\build.gradle'. > Configure project :x-pack:plugin:ml:cpp-snapshot Evaluating project ':x-pack:plugin:ml:cpp-snapshot' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\ml\cpp-snapshot\build.gradle'. > Configure project :x-pack:plugin:sql:jdbc Evaluating project ':x-pack:plugin:sql:jdbc' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\sql\jdbc\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:plugin:sql:sql-cli Evaluating project ':x-pack:plugin:sql:sql-cli' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\sql\sql-cli\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:plugin:sql:sql-proto Evaluating project ':x-pack:plugin:sql:sql-proto' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\sql\sql-proto\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:plugin:sql:sql-shared-client Evaluating project ':x-pack:plugin:sql:sql-shared-client' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\sql\sql-shared-client\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:full-cluster-restart:with-system-key Evaluating project ':x-pack:qa:full-cluster-restart:with-system-key' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\full-cluster-restart\with-system-key\build.gradle'. > Configure project :x-pack:qa:full-cluster-restart:without-system-key Evaluating project ':x-pack:qa:full-cluster-restart:without-system-key' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\full-cluster-restart\without-system-key\build.gradle'. > Configure project :x-pack:qa:rolling-upgrade:with-system-key Evaluating project ':x-pack:qa:rolling-upgrade:with-system-key' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\rolling-upgrade\with-system-key\build.gradle'. > Configure project :x-pack:qa:rolling-upgrade:without-system-key Evaluating project ':x-pack:qa:rolling-upgrade:without-system-key' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\rolling-upgrade\without-system-key\build.gradle'. > Configure project :x-pack:qa:sql:multinode Evaluating project ':x-pack:qa:sql:multinode' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\sql\multinode\build.gradle'. > Configure project :x-pack:qa:sql:no-security Evaluating project ':x-pack:qa:sql:no-security' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\sql\no-security\build.gradle'. > Configure project :x-pack:qa:sql:security Evaluating project ':x-pack:qa:sql:security' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\sql\security\build.gradle'. > Configure project :x-pack:qa:third-party:active-directory Evaluating project ':x-pack:qa:third-party:active-directory' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\third-party\active-directory\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:third-party:hipchat Evaluating project ':x-pack:qa:third-party:hipchat' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\third-party\hipchat\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:third-party:jira Evaluating project ':x-pack:qa:third-party:jira' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\third-party\jira\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:third-party:pagerduty Evaluating project ':x-pack:qa:third-party:pagerduty' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\third-party\pagerduty\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:third-party:slack Evaluating project ':x-pack:qa:third-party:slack' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\third-party\slack\build.gradle'. You are running forbidden-apis in the Gradle Daemon; disabling classloading cache to work around resource leak. > Configure project :x-pack:qa:sql:security:no-ssl Evaluating project ':x-pack:qa:sql:security:no-ssl' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\sql\security\no-ssl\build.gradle'. > Configure project :x-pack:qa:sql:security:ssl Evaluating project ':x-pack:qa:sql:security:ssl' using build file 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\qa\sql\security\ssl\build.gradle'. All projects evaluated. Selected primary task 'build' from project : Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-grouping/7.4.0-snapshot-1ed95c097b/lucene-grouping-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-core/7.4.0-snapshot-1ed95c097b/lucene-core-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-backward-codecs/7.4.0-snapshot-1ed95c097b/lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-analyzers-common/7.4.0-snapshot-1ed95c097b/lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-core/7.4.0-snapshot-1ed95c097b/lucene-core-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-analyzers-common/7.4.0-snapshot-1ed95c097b/lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-grouping/7.4.0-snapshot-1ed95c097b/lucene-grouping-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-backward-codecs/7.4.0-snapshot-1ed95c097b/lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-core/7.4.0-snapshot-1ed95c097b/lucene-core-7.4.0-snapshot-1ed95c097b.pom Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-backward-codecs/7.4.0-snapshot-1ed95c097b/lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.pom Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-grouping/7.4.0-snapshot-1ed95c097b/lucene-grouping-7.4.0-snapshot-1ed95c097b.pom Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-analyzers-common/7.4.0-snapshot-1ed95c097b/lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.pom Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-parent/7.4.0-snapshot-1ed95c097b/lucene-parent-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-parent/7.4.0-snapshot-1ed95c097b/lucene-parent-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-parent/7.4.0-snapshot-1ed95c097b/lucene-parent-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-parent/7.4.0-snapshot-1ed95c097b/lucene-parent-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-parent/7.4.0-snapshot-1ed95c097b/lucene-parent-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-parent/7.4.0-snapshot-1ed95c097b/lucene-parent-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-parent/7.4.0-snapshot-1ed95c097b/lucene-parent-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-parent/7.4.0-snapshot-1ed95c097b/lucene-parent-7.4.0-snapshot-1ed95c097b.jar] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-parent/7.4.0-snapshot-1ed95c097b/lucene-parent-7.4.0-snapshot-1ed95c097b.pom Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-solr-grandparent/7.4.0-snapshot-1ed95c097b/lucene-solr-grandparent-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-solr-grandparent/7.4.0-snapshot-1ed95c097b/lucene-solr-grandparent-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-solr-grandparent/7.4.0-snapshot-1ed95c097b/lucene-solr-grandparent-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-solr-grandparent/7.4.0-snapshot-1ed95c097b/lucene-solr-grandparent-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-solr-grandparent/7.4.0-snapshot-1ed95c097b/lucene-solr-grandparent-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-solr-grandparent/7.4.0-snapshot-1ed95c097b/lucene-solr-grandparent-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-solr-grandparent/7.4.0-snapshot-1ed95c097b/lucene-solr-grandparent-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-solr-grandparent/7.4.0-snapshot-1ed95c097b/lucene-solr-grandparent-7.4.0-snapshot-1ed95c097b.jar] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-solr-grandparent/7.4.0-snapshot-1ed95c097b/lucene-solr-grandparent-7.4.0-snapshot-1ed95c097b.pom Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-join/7.4.0-snapshot-1ed95c097b/lucene-join-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-join/7.4.0-snapshot-1ed95c097b/lucene-join-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-memory/7.4.0-snapshot-1ed95c097b/lucene-memory-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-misc/7.4.0-snapshot-1ed95c097b/lucene-misc-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-memory/7.4.0-snapshot-1ed95c097b/lucene-memory-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-misc/7.4.0-snapshot-1ed95c097b/lucene-misc-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-queries/7.4.0-snapshot-1ed95c097b/lucene-queries-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-queries/7.4.0-snapshot-1ed95c097b/lucene-queries-7.4.0-snapshot-1ed95c097b.jar] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-misc/7.4.0-snapshot-1ed95c097b/lucene-misc-7.4.0-snapshot-1ed95c097b.pom Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-memory/7.4.0-snapshot-1ed95c097b/lucene-memory-7.4.0-snapshot-1ed95c097b.pom Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-join/7.4.0-snapshot-1ed95c097b/lucene-join-7.4.0-snapshot-1ed95c097b.pom Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-queryparser/7.4.0-snapshot-1ed95c097b/lucene-queryparser-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-sandbox/7.4.0-snapshot-1ed95c097b/lucene-sandbox-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-queryparser/7.4.0-snapshot-1ed95c097b/lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-sandbox/7.4.0-snapshot-1ed95c097b/lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-spatial/7.4.0-snapshot-1ed95c097b/lucene-spatial-7.4.0-snapshot-1ed95c097b.pom] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-queryparser/7.4.0-snapshot-1ed95c097b/lucene-queryparser-7.4.0-snapshot-1ed95c097b.pom Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-spatial/7.4.0-snapshot-1ed95c097b/lucene-spatial-7.4.0-snapshot-1ed95c097b.jar] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-sandbox/7.4.0-snapshot-1ed95c097b/lucene-sandbox-7.4.0-snapshot-1ed95c097b.pom Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-spatial-extras/7.4.0-snapshot-1ed95c097b/lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-spatial-extras/7.4.0-snapshot-1ed95c097b/lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-spatial3d/7.4.0-snapshot-1ed95c097b/lucene-spatial3d-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-spatial3d/7.4.0-snapshot-1ed95c097b/lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-spatial-extras/7.4.0-snapshot-1ed95c097b/lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.pom Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-spatial/7.4.0-snapshot-1ed95c097b/lucene-spatial-7.4.0-snapshot-1ed95c097b.pom Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-spatial3d/7.4.0-snapshot-1ed95c097b/lucene-spatial3d-7.4.0-snapshot-1ed95c097b.pom Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-suggest/7.4.0-snapshot-1ed95c097b/lucene-suggest-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-suggest/7.4.0-snapshot-1ed95c097b/lucene-suggest-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-highlighter/7.4.0-snapshot-1ed95c097b/lucene-highlighter-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-highlighter/7.4.0-snapshot-1ed95c097b/lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-suggest/7.4.0-snapshot-1ed95c097b/lucene-suggest-7.4.0-snapshot-1ed95c097b.pom Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-highlighter/7.4.0-snapshot-1ed95c097b/lucene-highlighter-7.4.0-snapshot-1ed95c097b.pom Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-queries/7.4.0-snapshot-1ed95c097b/lucene-queries-7.4.0-snapshot-1ed95c097b.pom Starting process 'command 'C:\Users\jenkins\.java\java10\bin\java.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility Command: C:\Users\jenkins\.java\java10\bin\java.exe -version Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\java.exe'' Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-test-framework/7.4.0-snapshot-1ed95c097b/lucene-test-framework-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-codecs/7.4.0-snapshot-1ed95c097b/lucene-codecs-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-test-framework/7.4.0-snapshot-1ed95c097b/lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-codecs/7.4.0-snapshot-1ed95c097b/lucene-codecs-7.4.0-snapshot-1ed95c097b.jar] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-test-framework/7.4.0-snapshot-1ed95c097b/lucene-test-framework-7.4.0-snapshot-1ed95c097b.pom Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-codecs/7.4.0-snapshot-1ed95c097b/lucene-codecs-7.4.0-snapshot-1ed95c097b.pom Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-analyzers-phonetic/7.4.0-snapshot-1ed95c097b/lucene-analyzers-phonetic-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-analyzers-phonetic/7.4.0-snapshot-1ed95c097b/lucene-analyzers-phonetic-7.4.0-snapshot-1ed95c097b.jar] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-analyzers-phonetic/7.4.0-snapshot-1ed95c097b/lucene-analyzers-phonetic-7.4.0-snapshot-1ed95c097b.pom Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-analyzers-smartcn/7.4.0-snapshot-1ed95c097b/lucene-analyzers-smartcn-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-analyzers-smartcn/7.4.0-snapshot-1ed95c097b/lucene-analyzers-smartcn-7.4.0-snapshot-1ed95c097b.jar] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-analyzers-smartcn/7.4.0-snapshot-1ed95c097b/lucene-analyzers-smartcn-7.4.0-snapshot-1ed95c097b.pom Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-analyzers-kuromoji/7.4.0-snapshot-1ed95c097b/lucene-analyzers-kuromoji-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-analyzers-kuromoji/7.4.0-snapshot-1ed95c097b/lucene-analyzers-kuromoji-7.4.0-snapshot-1ed95c097b.jar] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-analyzers-kuromoji/7.4.0-snapshot-1ed95c097b/lucene-analyzers-kuromoji-7.4.0-snapshot-1ed95c097b.pom Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-analyzers-morfologik/7.4.0-snapshot-1ed95c097b/lucene-analyzers-morfologik-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-analyzers-morfologik/7.4.0-snapshot-1ed95c097b/lucene-analyzers-morfologik-7.4.0-snapshot-1ed95c097b.jar] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-analyzers-morfologik/7.4.0-snapshot-1ed95c097b/lucene-analyzers-morfologik-7.4.0-snapshot-1ed95c097b.pom Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-analyzers-stempel/7.4.0-snapshot-1ed95c097b/lucene-analyzers-stempel-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-analyzers-stempel/7.4.0-snapshot-1ed95c097b/lucene-analyzers-stempel-7.4.0-snapshot-1ed95c097b.jar] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-analyzers-stempel/7.4.0-snapshot-1ed95c097b/lucene-analyzers-stempel-7.4.0-snapshot-1ed95c097b.pom Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-expressions/7.4.0-snapshot-1ed95c097b/lucene-expressions-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-expressions/7.4.0-snapshot-1ed95c097b/lucene-expressions-7.4.0-snapshot-1ed95c097b.jar] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-expressions/7.4.0-snapshot-1ed95c097b/lucene-expressions-7.4.0-snapshot-1ed95c097b.pom Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-analyzers-nori/7.4.0-snapshot-1ed95c097b/lucene-analyzers-nori-7.4.0-snapshot-1ed95c097b.pom] Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-analyzers-nori/7.4.0-snapshot-1ed95c097b/lucene-analyzers-nori-7.4.0-snapshot-1ed95c097b.jar] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-analyzers-nori/7.4.0-snapshot-1ed95c097b/lucene-analyzers-nori-7.4.0-snapshot-1ed95c097b.pom Resource missing. [HTTP GET: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-analyzers-icu/7.4.0-snapshot-1ed95c097b/lucene-analyzers-icu-7.4.0-snapshot-1ed95c097b.pom] Download https://repo.maven.apache.org/maven2/com/ibm/icu/icu4j/61.1/icu4j-61.1.pom Resource missing. [HTTP HEAD: https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-analyzers-icu/7.4.0-snapshot-1ed95c097b/lucene-analyzers-icu-7.4.0-snapshot-1ed95c097b.jar] Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-analyzers-icu/7.4.0-snapshot-1ed95c097b/lucene-analyzers-icu-7.4.0-snapshot-1ed95c097b.pom Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-backward-codecs/7.4.0-snapshot-1ed95c097b/lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-highlighter/7.4.0-snapshot-1ed95c097b/lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-analyzers-common/7.4.0-snapshot-1ed95c097b/lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-core/7.4.0-snapshot-1ed95c097b/lucene-core-7.4.0-snapshot-1ed95c097b.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-join/7.4.0-snapshot-1ed95c097b/lucene-join-7.4.0-snapshot-1ed95c097b.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-memory/7.4.0-snapshot-1ed95c097b/lucene-memory-7.4.0-snapshot-1ed95c097b.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-misc/7.4.0-snapshot-1ed95c097b/lucene-misc-7.4.0-snapshot-1ed95c097b.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-queries/7.4.0-snapshot-1ed95c097b/lucene-queries-7.4.0-snapshot-1ed95c097b.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-queryparser/7.4.0-snapshot-1ed95c097b/lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-sandbox/7.4.0-snapshot-1ed95c097b/lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-spatial/7.4.0-snapshot-1ed95c097b/lucene-spatial-7.4.0-snapshot-1ed95c097b.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-spatial-extras/7.4.0-snapshot-1ed95c097b/lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-spatial3d/7.4.0-snapshot-1ed95c097b/lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-suggest/7.4.0-snapshot-1ed95c097b/lucene-suggest-7.4.0-snapshot-1ed95c097b.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-test-framework/7.4.0-snapshot-1ed95c097b/lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-codecs/7.4.0-snapshot-1ed95c097b/lucene-codecs-7.4.0-snapshot-1ed95c097b.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-grouping/7.4.0-snapshot-1ed95c097b/lucene-grouping-7.4.0-snapshot-1ed95c097b.jar Resource missing. [HTTP GET: http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/elasticsearch/plugin/x-pack/6.2.5-SNAPSHOT/maven-metadata.xml] Resource missing. [HTTP HEAD: http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/elasticsearch/plugin/x-pack/6.2.5-SNAPSHOT/x-pack-6.2.5-SNAPSHOT.pom] Resource missing. [HTTP HEAD: http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/elasticsearch/plugin/x-pack/6.2.5-SNAPSHOT/x-pack-6.2.5-SNAPSHOT.zip] Resource missing. [HTTP GET: https://snapshots.elastic.co/maven/org/elasticsearch/plugin/x-pack/6.2.5-SNAPSHOT/maven-metadata.xml] Cached resource https://snapshots.elastic.co/maven/org/elasticsearch/plugin/x-pack/6.2.5-SNAPSHOT/x-pack-6.2.5-SNAPSHOT.pom is up-to-date (lastModified: Wed Apr 25 18:21:18 UTC 2018). Tasks to be executed: [task ':libs:elasticsearch-core:compileJava', task ':libs:elasticsearch-core:processResources', task ':libs:elasticsearch-core:classes', task ':libs:elasticsearch-core:compileJava9Java', task ':libs:elasticsearch-core:processJava9Resources', task ':libs:elasticsearch-core:java9Classes', task ':libs:elasticsearch-core:jar', task ':libs:plugin-classloader:compileJava', task ':libs:plugin-classloader:processResources', task ':libs:plugin-classloader:classes', task ':libs:plugin-classloader:jar', task ':libs:secure-sm:compileJava', task ':libs:secure-sm:processResources', task ':libs:secure-sm:classes', task ':libs:secure-sm:jar', task ':libs:x-content:compileJava', task ':libs:x-content:processResources', task ':libs:x-content:classes', task ':libs:x-content:jar', task ':server:cli:compileJava', task ':server:cli:processResources', task ':server:cli:classes', task ':server:cli:jar', task ':server:compileJava', task ':server:generateModulesList', task ':server:generatePluginsList', task ':server:processResources', task ':server:classes', task ':server:compileJava9Java', task ':server:processJava9Resources', task ':server:java9Classes', task ':server:jar', task ':benchmarks:compileJava', task ':benchmarks:processResources', task ':benchmarks:classes', task ':benchmarks:copyCheckstyleConf', task ':benchmarks:checkstyleMain', task ':benchmarks:compileTestJava', task ':benchmarks:processTestResources', task ':benchmarks:testClasses', task ':benchmarks:checkstyleTest', task ':benchmarks:forbiddenApisMain', task ':benchmarks:forbiddenApisTest', task ':benchmarks:forbiddenApis', task ':benchmarks:checkstyle', task ':benchmarks:dependencyLicenses', task ':benchmarks:filepermissions', task ':benchmarks:forbiddenPatterns', task ':benchmarks:jarHell', task ':benchmarks:licenseHeaders', task ':test:logger-usage:compileJava', task ':test:logger-usage:processResources', task ':test:logger-usage:classes', task ':test:logger-usage:jar', task ':benchmarks:loggerUsageCheck', task ':build-tools:compileJava', task ':build-tools:compileGroovy', task ':build-tools:writeVersionProperties', task ':build-tools:processResources', task ':build-tools:classes', task ':build-tools:jar', task ':benchmarks:namingConventions', task ':benchmarks:thirdPartyAudit', task ':benchmarks:precommit', task ':build-tools:copyCheckstyleConf', task ':build-tools:checkstyleMain', task ':build-tools:compileTestJava', task ':build-tools:compileTestGroovy', task ':build-tools:processTestResources', task ':build-tools:testClasses', task ':build-tools:checkstyleTest', task ':build-tools:checkstyle', task ':build-tools:dependencyLicenses', task ':build-tools:filepermissions', task ':build-tools:forbiddenApisMain', task ':build-tools:forbiddenApisTest', task ':build-tools:forbiddenApis', task ':build-tools:forbiddenPatterns', task ':build-tools:jarHell', task ':build-tools:licenseHeaders', task ':build-tools:namingConventions', task ':build-tools:namingConventionsMain', task ':build-tools:thirdPartyAudit', task ':build-tools:precommit', task ':build-tools:test', task ':server:copyCheckstyleConf', task ':server:checkstyleJava9', task ':server:checkstyleMain', task ':client:rest:compileJava', task ':client:rest:processResources', task ':client:rest:classes', task ':client:rest:jar', task ':libs:elasticsearch-nio:compileJava', task ':libs:elasticsearch-nio:processResources', task ':libs:elasticsearch-nio:classes', task ':libs:elasticsearch-nio:jar', task ':test:framework:compileJava', task ':test:framework:processResources', task ':test:framework:classes', task ':test:framework:jar', task ':server:compileTestJava', task ':server:processTestResources', task ':server:testClasses', task ':server:checkstyleTest', task ':server:checkstyle', task ':server:dependencyLicenses', task ':server:filepermissions', task ':server:forbiddenApisMain', task ':server:forbiddenApisTest', task ':server:forbiddenApis', task ':server:forbiddenPatterns', task ':server:jarHell', task ':server:licenseHeaders', task ':server:loggerUsageCheck', task ':server:namingConventions', task ':server:thirdPartyAudit', task ':server:precommit', task ':libs:elasticsearch-core:copyCheckstyleConf', task ':libs:elasticsearch-core:checkstyleJava9', task ':libs:elasticsearch-core:checkstyleMain', task ':libs:elasticsearch-core:compileTestJava', task ':libs:elasticsearch-core:processTestResources', task ':libs:elasticsearch-core:testClasses', task ':libs:elasticsearch-core:checkstyleTest', task ':libs:elasticsearch-core:checkstyle', task ':libs:elasticsearch-core:dependencyLicenses', task ':libs:elasticsearch-core:filepermissions', task ':libs:elasticsearch-core:forbiddenApisMain', task ':libs:elasticsearch-core:forbiddenApisTest', task ':libs:elasticsearch-core:forbiddenApis', task ':libs:elasticsearch-core:forbiddenPatterns', task ':libs:elasticsearch-core:jarHell', task ':libs:elasticsearch-core:licenseHeaders', task ':libs:elasticsearch-core:loggerUsageCheck', task ':libs:elasticsearch-core:namingConventions', task ':libs:elasticsearch-core:thirdPartyAudit', task ':libs:elasticsearch-core:precommit', task ':test:framework:copyCheckstyleConf', task ':test:framework:checkstyleMain', task ':test:framework:compileTestJava', task ':test:framework:processTestResources', task ':test:framework:testClasses', task ':test:framework:checkstyleTest', task ':test:framework:checkstyle', task ':test:framework:dependencyLicenses', task ':test:framework:filepermissions', task ':test:framework:forbiddenApisMain', task ':test:framework:forbiddenApisTest', task ':test:framework:forbiddenApis', task ':test:framework:forbiddenPatterns', task ':test:framework:jarHell', task ':test:framework:licenseHeaders', task ':test:framework:loggerUsageCheck', task ':test:framework:namingConventions', task ':test:framework:namingConventionsMain', task ':test:framework:thirdPartyAudit', task ':test:framework:precommit', task ':test:framework:test', task ':test:logger-usage:copyCheckstyleConf', task ':test:logger-usage:checkstyleMain', task ':test:logger-usage:compileTestJava', task ':test:logger-usage:processTestResources', task ':test:logger-usage:testClasses', task ':test:logger-usage:checkstyleTest', task ':test:logger-usage:checkstyle', task ':test:logger-usage:dependencyLicenses', task ':test:logger-usage:filepermissions', task ':test:logger-usage:forbiddenApisMain', task ':test:logger-usage:forbiddenApisTest', task ':test:logger-usage:forbiddenApis', task ':test:logger-usage:forbiddenPatterns', task ':test:logger-usage:jarHell', task ':test:logger-usage:licenseHeaders', task ':test:logger-usage:loggerUsageCheck', task ':test:logger-usage:namingConventions', task ':test:logger-usage:thirdPartyAudit', task ':test:logger-usage:precommit', task ':test:logger-usage:test', task ':libs:elasticsearch-core:test', task ':libs:plugin-classloader:copyCheckstyleConf', task ':libs:plugin-classloader:checkstyleMain', task ':libs:plugin-classloader:compileTestJava', task ':libs:plugin-classloader:processTestResources', task ':libs:plugin-classloader:testClasses', task ':libs:plugin-classloader:checkstyleTest', task ':libs:plugin-classloader:checkstyle', task ':libs:plugin-classloader:dependencyLicenses', task ':libs:plugin-classloader:filepermissions', task ':libs:plugin-classloader:forbiddenApisMain', task ':libs:plugin-classloader:forbiddenApisTest', task ':libs:plugin-classloader:forbiddenApis', task ':libs:plugin-classloader:forbiddenPatterns', task ':libs:plugin-classloader:jarHell', task ':libs:plugin-classloader:licenseHeaders', task ':libs:plugin-classloader:loggerUsageCheck', task ':libs:plugin-classloader:namingConventions', task ':libs:plugin-classloader:thirdPartyAudit', task ':libs:plugin-classloader:precommit', task ':libs:plugin-classloader:test', task ':libs:secure-sm:copyCheckstyleConf', task ':libs:secure-sm:checkstyleMain', task ':libs:secure-sm:compileTestJava', task ':libs:secure-sm:processTestResources', task ':libs:secure-sm:testClasses', task ':libs:secure-sm:checkstyleTest', task ':libs:secure-sm:checkstyle', task ':libs:secure-sm:dependencyLicenses', task ':libs:secure-sm:filepermissions', task ':libs:secure-sm:forbiddenApisMain', task ':libs:secure-sm:forbiddenApisTest', task ':libs:secure-sm:forbiddenApis', task ':libs:secure-sm:forbiddenPatterns', task ':libs:secure-sm:jarHell', task ':libs:secure-sm:licenseHeaders', task ':libs:secure-sm:loggerUsageCheck', task ':libs:secure-sm:namingConventions', task ':libs:secure-sm:thirdPartyAudit', task ':libs:secure-sm:precommit', task ':libs:secure-sm:test', task ':libs:x-content:copyCheckstyleConf', task ':libs:x-content:checkstyleMain', task ':libs:x-content:compileTestJava', task ':libs:x-content:processTestResources', task ':libs:x-content:testClasses', task ':libs:x-content:checkstyleTest', task ':libs:x-content:checkstyle', task ':libs:x-content:dependencyLicenses', task ':libs:x-content:filepermissions', task ':libs:x-content:forbiddenApisMain', task ':libs:x-content:forbiddenApisTest', task ':libs:x-content:forbiddenApis', task ':libs:x-content:forbiddenPatterns', task ':libs:x-content:jarHell', task ':libs:x-content:licenseHeaders', task ':libs:x-content:loggerUsageCheck', task ':libs:x-content:namingConventions', task ':libs:x-content:thirdPartyAudit', task ':libs:x-content:precommit', task ':libs:x-content:test', task ':server:cli:copyCheckstyleConf', task ':server:cli:checkstyleMain', task ':server:cli:compileTestJava', task ':server:cli:processTestResources', task ':server:cli:testClasses', task ':server:cli:checkstyleTest', task ':server:cli:checkstyle', task ':server:cli:dependencyLicenses', task ':server:cli:filepermissions', task ':server:cli:forbiddenApisMain', task ':server:cli:forbiddenApisTest', task ':server:cli:forbiddenApis', task ':server:cli:forbiddenPatterns', task ':server:cli:jarHell', task ':server:cli:licenseHeaders', task ':server:cli:loggerUsageCheck', task ':server:cli:namingConventions', task ':server:cli:thirdPartyAudit', task ':server:cli:precommit', task ':server:cli:test', task ':server:test', task ':benchmarks:test', task ':benchmarks:jar', task ':benchmarks:extractJar', task ':benchmarks:verifyJarLicense', task ':benchmarks:verifyJarNotice', task ':libs:elasticsearch-core:javadoc', task ':libs:plugin-classloader:javadoc', task ':libs:secure-sm:javadoc', task ':libs:x-content:javadoc', task ':server:cli:javadoc', task ':server:javadoc', task ':benchmarks:javadoc', task ':benchmarks:javadocJar', task ':benchmarks:extractJavadocJar', task ':benchmarks:verifyJavadocJarLicense', task ':benchmarks:verifyJavadocJarNotice', task ':benchmarks:shadowJar', task ':benchmarks:extractShadowJar', task ':benchmarks:verifyShadowJarLicense', task ':benchmarks:verifyShadowJarNotice', task ':benchmarks:sourcesJar', task ':benchmarks:extractSourcesJar', task ':benchmarks:verifySourcesJarLicense', task ':benchmarks:verifySourcesJarNotice', task ':benchmarks:check', task ':benchmarks:build', task ':build-tools:generatePomFileForNebulaPublication', task ':build-tools:javadoc', task ':build-tools:javadocJar', task ':build-tools:sourcesJar', task ':build-tools:assemble', task ':build-tools:extractJar', task ':build-tools:verifyJarLicense', task ':build-tools:verifyJarNotice', task ':build-tools:extractJavadocJar', task ':build-tools:verifyJavadocJarLicense', task ':build-tools:verifyJavadocJarNotice', task ':build-tools:extractSourcesJar', task ':build-tools:verifySourcesJarLicense', task ':build-tools:verifySourcesJarNotice', task ':build-tools:check', task ':build-tools:build', task ':docs:buildRestTests', task ':docs:copyCheckstyleConf', task ':docs:compileTestJava', task ':docs:processTestResources', task ':docs:testClasses', task ':docs:checkstyleTest', task ':docs:forbiddenApisTest', task ':docs:forbiddenApis', task ':docs:integTestCluster#prepareCluster.cleanShared', task ':docs:integTestCluster#clean', task ':docs:integTestCluster#createCwd', task ':docs:integTestCluster#checkPrevious', task ':docs:integTestCluster#stopPrevious', task ':distribution:buildDefaultNotice', task ':modules:aggs-matrix-stats:compileJava', task ':modules:aggs-matrix-stats:processResources', task ':modules:aggs-matrix-stats:classes', task ':modules:aggs-matrix-stats:jar', task ':modules:aggs-matrix-stats:copyPluginPropertiesTemplate', task ':modules:aggs-matrix-stats:pluginProperties', task ':modules:aggs-matrix-stats:bundlePlugin', task ':modules:analysis-common:compileJava', task ':modules:analysis-common:processResources', task ':modules:analysis-common:classes', task ':modules:analysis-common:jar', task ':modules:analysis-common:copyPluginPropertiesTemplate', task ':modules:analysis-common:pluginProperties', task ':modules:analysis-common:bundlePlugin', task ':libs:grok:compileJava', task ':libs:grok:processResources', task ':libs:grok:classes', task ':libs:grok:jar', task ':modules:ingest-common:compileJava', task ':modules:ingest-common:processResources', task ':modules:ingest-common:classes', task ':modules:ingest-common:jar', task ':modules:ingest-common:copyPluginPropertiesTemplate', task ':modules:ingest-common:pluginProperties', task ':modules:ingest-common:bundlePlugin', task ':modules:lang-expression:compileJava', task ':modules:lang-expression:processResources', task ':modules:lang-expression:classes', task ':modules:lang-expression:jar', task ':modules:lang-expression:copyPluginPropertiesTemplate', task ':modules:lang-expression:pluginProperties', task ':modules:lang-expression:bundlePlugin', task ':modules:lang-mustache:compileJava', task ':modules:lang-mustache:processResources', task ':modules:lang-mustache:classes', task ':modules:lang-mustache:jar', task ':modules:lang-mustache:copyPluginPropertiesTemplate', task ':modules:lang-mustache:pluginProperties', task ':modules:lang-mustache:bundlePlugin', task ':modules:lang-painless:spi:compileJava', task ':modules:lang-painless:spi:processResources', task ':modules:lang-painless:spi:classes', task ':modules:lang-painless:spi:jar', task ':modules:lang-painless:compileJava', task ':modules:lang-painless:processResources', task ':modules:lang-painless:classes', task ':modules:lang-painless:jar', task ':modules:lang-painless:copyPluginPropertiesTemplate', task ':modules:lang-painless:pluginProperties', task ':modules:lang-painless:bundlePlugin', task ':modules:mapper-extras:compileJava', task ':modules:mapper-extras:processResources', task ':modules:mapper-extras:classes', task ':modules:mapper-extras:jar', task ':modules:mapper-extras:copyPluginPropertiesTemplate', task ':modules:mapper-extras:pluginProperties', task ':modules:mapper-extras:bundlePlugin', task ':modules:parent-join:compileJava', task ':modules:parent-join:processResources', task ':modules:parent-join:classes', task ':modules:parent-join:jar', task ':modules:parent-join:copyPluginPropertiesTemplate', task ':modules:parent-join:pluginProperties', task ':modules:parent-join:bundlePlugin', task ':modules:percolator:compileJava', task ':modules:percolator:processResources', task ':modules:percolator:classes', task ':modules:percolator:jar', task ':modules:percolator:copyPluginPropertiesTemplate', task ':modules:percolator:pluginProperties', task ':modules:percolator:bundlePlugin', task ':modules:rank-eval:compileJava', task ':modules:rank-eval:processResources', task ':modules:rank-eval:classes', task ':modules:rank-eval:jar', task ':modules:rank-eval:copyPluginPropertiesTemplate', task ':modules:rank-eval:pluginProperties', task ':modules:rank-eval:bundlePlugin', task ':modules:reindex:compileJava', task ':modules:reindex:processResources', task ':modules:reindex:classes', task ':modules:reindex:jar', task ':modules:reindex:copyPluginPropertiesTemplate', task ':modules:reindex:pluginProperties', task ':modules:reindex:bundlePlugin', task ':modules:repository-url:compileJava', task ':modules:repository-url:processResources', task ':modules:repository-url:classes', task ':modules:repository-url:jar', task ':modules:repository-url:copyPluginPropertiesTemplate', task ':modules:repository-url:pluginProperties', task ':modules:repository-url:bundlePlugin', task ':modules:transport-netty4:compileJava', task ':modules:transport-netty4:processResources', task ':modules:transport-netty4:classes', task ':modules:transport-netty4:jar', task ':modules:transport-netty4:copyPluginPropertiesTemplate', task ':modules:transport-netty4:pluginProperties', task ':modules:transport-netty4:bundlePlugin', task ':distribution:processOssOutputs', task ':distribution:buildOssBin', task ':distribution:buildOssConfig', task ':distribution:buildOssLog4jConfig', task ':distribution:buildOssModules', task ':distribution:archives:createLogsDir', task ':distribution:archives:createPluginsDir', task ':distribution:tools:launchers:compileJava', task ':distribution:tools:launchers:processResources', task ':distribution:tools:launchers:classes', task ':distribution:tools:launchers:jar', task ':distribution:tools:plugin-cli:compileJava', task ':distribution:tools:plugin-cli:processResources', task ':distribution:tools:plugin-cli:classes', task ':distribution:tools:plugin-cli:jar', task ':distribution:archives:buildOssZip', task ':docs:integTestCluster#extract', task ':plugins:analysis-icu:generateNotice', task ':plugins:analysis-icu:compileJava', task ':plugins:analysis-icu:processResources', task ':plugins:analysis-icu:classes', task ':plugins:analysis-icu:jar', task ':plugins:analysis-icu:copyPluginPropertiesTemplate', task ':plugins:analysis-icu:pluginProperties', task ':plugins:analysis-icu:bundlePlugin', task ':plugins:analysis-kuromoji:generateNotice', task ':plugins:analysis-kuromoji:compileJava', task ':plugins:analysis-kuromoji:processResources', task ':plugins:analysis-kuromoji:classes', task ':plugins:analysis-kuromoji:jar', task ':plugins:analysis-kuromoji:copyPluginPropertiesTemplate', task ':plugins:analysis-kuromoji:pluginProperties', task ':plugins:analysis-kuromoji:bundlePlugin', task ':plugins:analysis-nori:generateNotice', task ':plugins:analysis-nori:compileJava', task ':plugins:analysis-nori:processResources', task ':plugins:analysis-nori:classes', task ':plugins:analysis-nori:jar', task ':plugins:analysis-nori:copyPluginPropertiesTemplate', task ':plugins:analysis-nori:pluginProperties', task ':plugins:analysis-nori:bundlePlugin', task ':plugins:analysis-phonetic:generateNotice', task ':plugins:analysis-phonetic:compileJava', task ':plugins:analysis-phonetic:processResources', task ':plugins:analysis-phonetic:classes', task ':plugins:analysis-phonetic:jar', task ':plugins:analysis-phonetic:copyPluginPropertiesTemplate', task ':plugins:analysis-phonetic:pluginProperties', task ':plugins:analysis-phonetic:bundlePlugin', task ':plugins:analysis-smartcn:generateNotice', task ':plugins:analysis-smartcn:compileJava', task ':plugins:analysis-smartcn:processResources', task ':plugins:analysis-smartcn:classes', task ':plugins:analysis-smartcn:jar', task ':plugins:analysis-smartcn:copyPluginPropertiesTemplate', task ':plugins:analysis-smartcn:pluginProperties', task ':plugins:analysis-smartcn:bundlePlugin', task ':plugins:analysis-stempel:generateNotice', task ':plugins:analysis-stempel:compileJava', task ':plugins:analysis-stempel:processResources', task ':plugins:analysis-stempel:classes', task ':plugins:analysis-stempel:jar', task ':plugins:analysis-stempel:copyPluginPropertiesTemplate', task ':plugins:analysis-stempel:pluginProperties', task ':plugins:analysis-stempel:bundlePlugin', task ':plugins:analysis-ukrainian:generateNotice', task ':plugins:analysis-ukrainian:compileJava', task ':plugins:analysis-ukrainian:processResources', task ':plugins:analysis-ukrainian:classes', task ':plugins:analysis-ukrainian:jar', task ':plugins:analysis-ukrainian:copyPluginPropertiesTemplate', task ':plugins:analysis-ukrainian:pluginProperties', task ':plugins:analysis-ukrainian:bundlePlugin', task ':plugins:discovery-azure-classic:generateNotice', task ':plugins:discovery-azure-classic:compileJava', task ':plugins:discovery-azure-classic:processResources', task ':plugins:discovery-azure-classic:classes', task ':plugins:discovery-azure-classic:jar', task ':plugins:discovery-azure-classic:copyPluginPropertiesTemplate', task ':plugins:discovery-azure-classic:pluginProperties', task ':plugins:discovery-azure-classic:bundlePlugin', task ':plugins:discovery-ec2:generateNotice', task ':plugins:discovery-ec2:compileJava', task ':plugins:discovery-ec2:processResources', task ':plugins:discovery-ec2:classes', task ':plugins:discovery-ec2:jar', task ':plugins:discovery-ec2:copyPluginPropertiesTemplate', task ':plugins:discovery-ec2:pluginProperties', task ':plugins:discovery-ec2:bundlePlugin', task ':plugins:discovery-file:generateNotice', task ':plugins:discovery-file:compileJava', task ':plugins:discovery-file:processResources', task ':plugins:discovery-file:classes', task ':plugins:discovery-file:jar', task ':plugins:discovery-file:copyPluginPropertiesTemplate', task ':plugins:discovery-file:pluginProperties', task ':plugins:discovery-file:bundlePlugin', task ':plugins:discovery-gce:generateNotice', task ':plugins:discovery-gce:compileJava', task ':plugins:discovery-gce:processResources', task ':plugins:discovery-gce:classes', task ':plugins:discovery-gce:jar', task ':plugins:discovery-gce:copyPluginPropertiesTemplate', task ':plugins:discovery-gce:pluginProperties', task ':plugins:discovery-gce:bundlePlugin', task ':plugins:ingest-attachment:generateNotice', task ':plugins:ingest-attachment:compileJava', task ':plugins:ingest-attachment:processResources', task ':plugins:ingest-attachment:classes', task ':plugins:ingest-attachment:jar', task ':plugins:ingest-attachment:copyPluginPropertiesTemplate', task ':plugins:ingest-attachment:pluginProperties', task ':plugins:ingest-attachment:bundlePlugin', task ':plugins:ingest-geoip:copyDefaultGeoIp2DatabaseFiles', task ':plugins:ingest-geoip:generateNotice', task ':plugins:ingest-geoip:compileJava', task ':plugins:ingest-geoip:processResources', task ':plugins:ingest-geoip:classes', task ':plugins:ingest-geoip:jar', task ':plugins:ingest-geoip:copyPluginPropertiesTemplate', task ':plugins:ingest-geoip:pluginProperties', task ':plugins:ingest-geoip:bundlePlugin', task ':plugins:ingest-user-agent:generateNotice', task ':plugins:ingest-user-agent:compileJava', task ':plugins:ingest-user-agent:processResources', task ':plugins:ingest-user-agent:classes', task ':plugins:ingest-user-agent:jar', task ':plugins:ingest-user-agent:copyPluginPropertiesTemplate', task ':plugins:ingest-user-agent:pluginProperties', task ':plugins:ingest-user-agent:bundlePlugin', task ':plugins:mapper-murmur3:generateNotice', task ':plugins:mapper-murmur3:compileJava', task ':plugins:mapper-murmur3:processResources', task ':plugins:mapper-murmur3:classes', task ':plugins:mapper-murmur3:jar', task ':plugins:mapper-murmur3:copyPluginPropertiesTemplate', task ':plugins:mapper-murmur3:pluginProperties', task ':plugins:mapper-murmur3:bundlePlugin', task ':plugins:mapper-size:generateNotice', task ':plugins:mapper-size:compileJava', task ':plugins:mapper-size:processResources', task ':plugins:mapper-size:classes', task ':plugins:mapper-size:jar', task ':plugins:mapper-size:copyPluginPropertiesTemplate', task ':plugins:mapper-size:pluginProperties', task ':plugins:mapper-size:bundlePlugin', task ':plugins:store-smb:generateNotice', task ':plugins:store-smb:compileJava', task ':plugins:store-smb:processResources', task ':plugins:store-smb:classes', task ':plugins:store-smb:jar', task ':plugins:store-smb:copyPluginPropertiesTemplate', task ':plugins:store-smb:pluginProperties', task ':plugins:store-smb:bundlePlugin', task ':plugins:transport-nio:generateNotice', task ':plugins:transport-nio:compileJava', task ':plugins:transport-nio:processResources', task ':plugins:transport-nio:classes', task ':plugins:transport-nio:jar', task ':plugins:transport-nio:copyPluginPropertiesTemplate', task ':plugins:transport-nio:pluginProperties', task ':plugins:transport-nio:bundlePlugin', task ':docs:integTestCluster#configure', task ':docs:integTestCluster#copyPlugins', task ':docs:integTestCluster#installAnalysisIcuPlugin', task ':docs:integTestCluster#installAnalysisKuromojiPlugin', task ':docs:integTestCluster#installAnalysisNoriPlugin', task ':docs:integTestCluster#installAnalysisPhoneticPlugin', task ':docs:integTestCluster#installAnalysisSmartcnPlugin', task ':docs:integTestCluster#installAnalysisStempelPlugin', task ':docs:integTestCluster#installAnalysisUkrainianPlugin', task ':docs:integTestCluster#installDiscoveryAzureClassicPlugin', task ':docs:integTestCluster#installDiscoveryEc2Plugin', task ':docs:integTestCluster#installDiscoveryFilePlugin', task ':docs:integTestCluster#installDiscoveryGcePlugin', task ':docs:integTestCluster#installIngestAttachmentPlugin', task ':docs:integTestCluster#installIngestGeoipPlugin', task ':docs:integTestCluster#installIngestUserAgentPlugin', task ':docs:integTestCluster#installMapperMurmur3Plugin', task ':docs:integTestCluster#installMapperSizePlugin', task ':docs:integTestCluster#installStoreSmbPlugin', task ':docs:integTestCluster#installTransportNioPlugin', task ':docs:integTestCluster#extraConfig', task ':docs:integTestCluster#start', task ':rest-api-spec:compileJava', task ':rest-api-spec:processResources', task ':rest-api-spec:classes', task ':rest-api-spec:jar', task ':docs:copyRestSpec', task ':docs:checkstyle', task ':docs:filepermissions', task ':docs:forbiddenPatterns', task ':docs:jarHell', task ':docs:licenseHeaders', task ':docs:loggerUsageCheck', task ':docs:namingConventions', task ':docs:thirdPartyAudit', task ':docs:precommit', task ':x-pack:plugin:copyPluginPropertiesTemplate', task ':x-pack:plugin:pluginProperties', task ':x-pack:plugin:core:generateNotice', task ':x-pack:plugin:core:compileJava', task ':x-pack:plugin:core:processResources', task ':x-pack:plugin:core:classes', task ':x-pack:plugin:core:jar', task ':x-pack:plugin:core:copyPluginPropertiesTemplate', task ':x-pack:plugin:core:pluginProperties', task ':x-pack:plugin:core:bundlePlugin', task ':x-pack:plugin:deprecation:generateNotice', task ':x-pack:plugin:deprecation:compileJava', task ':x-pack:plugin:deprecation:processResources', task ':x-pack:plugin:deprecation:classes', task ':x-pack:plugin:deprecation:jar', task ':x-pack:plugin:deprecation:copyPluginPropertiesTemplate', task ':x-pack:plugin:deprecation:pluginProperties', task ':x-pack:plugin:deprecation:bundlePlugin', task ':x-pack:plugin:graph:generateNotice', task ':x-pack:plugin:graph:compileJava', task ':x-pack:plugin:graph:processResources', task ':x-pack:plugin:graph:classes', task ':x-pack:plugin:graph:jar', task ':x-pack:plugin:graph:copyPluginPropertiesTemplate', task ':x-pack:plugin:graph:pluginProperties', task ':x-pack:plugin:graph:bundlePlugin', task ':x-pack:plugin:logstash:generateNotice', task ':x-pack:plugin:logstash:compileJava', task ':x-pack:plugin:logstash:processResources', task ':x-pack:plugin:logstash:classes', task ':x-pack:plugin:logstash:jar', task ':x-pack:plugin:logstash:copyPluginPropertiesTemplate', task ':x-pack:plugin:logstash:pluginProperties', task ':x-pack:plugin:logstash:bundlePlugin', task ':x-pack:plugin:ml:cpp-snapshot:downloadMachineLearningSnapshot', task ':x-pack:plugin:ml:extractNativeLicenses', task ':x-pack:plugin:ml:generateNotice', task ':x-pack:plugin:ml:compileJava', task ':x-pack:plugin:ml:processResources', task ':x-pack:plugin:ml:classes', task ':x-pack:plugin:ml:jar', task ':x-pack:plugin:ml:copyPluginPropertiesTemplate', task ':x-pack:plugin:ml:pluginProperties', task ':x-pack:plugin:ml:bundlePlugin', task ':client:sniffer:compileJava', task ':client:sniffer:processResources', task ':client:sniffer:classes', task ':client:sniffer:jar', task ':x-pack:plugin:monitoring:generateNotice', task ':x-pack:plugin:monitoring:compileJava', task ':x-pack:plugin:monitoring:processResources', task ':x-pack:plugin:monitoring:classes', task ':x-pack:plugin:monitoring:jar', task ':x-pack:plugin:monitoring:copyPluginPropertiesTemplate', task ':x-pack:plugin:monitoring:pluginProperties', task ':x-pack:plugin:monitoring:bundlePlugin', task ':x-pack:plugin:rollup:generateNotice', task ':x-pack:plugin:rollup:compileJava', task ':x-pack:plugin:rollup:processResources', task ':x-pack:plugin:rollup:classes', task ':x-pack:plugin:rollup:jar', task ':x-pack:plugin:rollup:copyPluginPropertiesTemplate', task ':x-pack:plugin:rollup:pluginProperties', task ':x-pack:plugin:rollup:bundlePlugin', task ':x-pack:plugin:security:generateNotice', task ':x-pack:plugin:security:compileJava', task ':x-pack:plugin:security:processResources', task ':x-pack:plugin:security:classes', task ':x-pack:plugin:security:jar', task ':x-pack:plugin:security:copyPluginPropertiesTemplate', task ':x-pack:plugin:security:pluginProperties', task ':x-pack:plugin:security:bundlePlugin', task ':x-pack:plugin:sql:generateNotice', task ':x-pack:plugin:sql:sql-proto:compileJava', task ':x-pack:plugin:sql:sql-proto:processResources', task ':x-pack:plugin:sql:sql-proto:classes', task ':x-pack:plugin:sql:sql-proto:jar', task ':x-pack:plugin:sql:compileJava', task ':x-pack:plugin:sql:processResources', task ':x-pack:plugin:sql:classes', task ':x-pack:plugin:sql:jar', task ':x-pack:plugin:sql:copyPluginPropertiesTemplate', task ':x-pack:plugin:sql:pluginProperties', task ':x-pack:plugin:sql:sql-shared-client:compileJava', task ':x-pack:plugin:sql:sql-shared-client:processResources', task ':x-pack:plugin:sql:sql-shared-client:classes', task ':x-pack:plugin:sql:sql-shared-client:jar', task ':x-pack:plugin:sql:sql-cli:compileJava', task ':x-pack:plugin:sql:sql-cli:processResources', task ':x-pack:plugin:sql:sql-cli:classes', task ':x-pack:plugin:sql:sql-cli:jar', task ':x-pack:plugin:sql:bundlePlugin', task ':x-pack:plugin:upgrade:generateNotice', task ':x-pack:plugin:upgrade:compileJava', task ':x-pack:plugin:upgrade:processResources', task ':x-pack:plugin:upgrade:classes', task ':x-pack:plugin:upgrade:jar', task ':x-pack:plugin:upgrade:copyPluginPropertiesTemplate', task ':x-pack:plugin:upgrade:pluginProperties', task ':x-pack:plugin:upgrade:bundlePlugin', task ':x-pack:plugin:watcher:generateNotice', task ':x-pack:plugin:watcher:compileJava', task ':x-pack:plugin:watcher:processResources', task ':x-pack:plugin:watcher:classes', task ':x-pack:plugin:watcher:jar', task ':x-pack:plugin:watcher:copyPluginPropertiesTemplate', task ':x-pack:plugin:watcher:pluginProperties', task ':x-pack:plugin:watcher:bundlePlugin', task ':x-pack:plugin:bundlePlugin', task ':distribution:processDefaultOutputs', task ':distribution:buildDefaultBin', task ':distribution:buildDefaultConfig', task ':distribution:buildDefaultLog4jConfig', task ':distribution:processTransportOutputs', task ':distribution:buildTransportModules', task ':distribution:archives:buildIntegTestZip', task ':plugins:analysis-icu:integTestCluster#prepareCluster.cleanShared', task ':plugins:analysis-icu:integTestCluster#clean', task ':plugins:analysis-icu:integTestCluster#createCwd', task ':plugins:analysis-icu:integTestCluster#checkPrevious', task ':plugins:analysis-icu:integTestCluster#stopPrevious', task ':plugins:analysis-icu:integTestCluster#extract', task ':plugins:analysis-icu:integTestCluster#configure', task ':plugins:analysis-icu:integTestCluster#copyPlugins', task ':plugins:analysis-icu:integTestCluster#installAnalysisIcuPlugin', task ':plugins:analysis-icu:integTestCluster#start', task ':plugins:analysis-icu:processTestResources', task ':plugins:analysis-icu:copyRestSpec', task ':server:integTest', task ':plugins:analysis-icu:copyCheckstyleConf', task ':plugins:analysis-icu:checkstyleMain', task ':plugins:analysis-icu:compileTestJava', task ':plugins:analysis-icu:testClasses', task ':plugins:analysis-icu:checkstyleTest', task ':plugins:analysis-icu:checkstyle', task ':plugins:analysis-icu:dependencyLicenses', task ':plugins:analysis-icu:filepermissions', task ':plugins:analysis-icu:forbiddenApisMain', task ':plugins:analysis-icu:forbiddenApisTest', task ':plugins:analysis-icu:forbiddenApis', task ':plugins:analysis-icu:forbiddenPatterns', task ':plugins:analysis-icu:jarHell', task ':plugins:analysis-icu:licenseHeaders', task ':plugins:analysis-icu:loggerUsageCheck', task ':plugins:analysis-icu:namingConventions', task ':plugins:analysis-icu:thirdPartyAudit', task ':plugins:analysis-icu:precommit', task ':rest-api-spec:copyCheckstyleConf', task ':rest-api-spec:checkstyleMain', task ':rest-api-spec:compileTestJava', task ':rest-api-spec:processTestResources', task ':rest-api-spec:testClasses', task ':rest-api-spec:checkstyleTest', task ':rest-api-spec:checkstyle', task ':rest-api-spec:dependencyLicenses', task ':rest-api-spec:filepermissions', task ':rest-api-spec:forbiddenApisMain', task ':rest-api-spec:forbiddenApisTest', task ':rest-api-spec:forbiddenApis', task ':rest-api-spec:forbiddenPatterns', task ':rest-api-spec:jarHell', task ':rest-api-spec:licenseHeaders', task ':rest-api-spec:loggerUsageCheck', task ':rest-api-spec:namingConventions', task ':rest-api-spec:thirdPartyAudit', task ':rest-api-spec:precommit', task ':rest-api-spec:test', task ':plugins:analysis-icu:test', task ':distribution:archives:integ-test-zip:distTar', task ':distribution:archives:integ-test-zip:distZip', task ':distribution:archives:integ-test-zip:generatePomFileForNebulaPublication', task ':distribution:archives:integ-test-zip:generatePomFileForNebulaRealPomPublication', task ':distribution:archives:integ-test-zip:assemble', task ':distribution:archives:integ-test-zip:integTestCluster#prepareCluster.cleanShared', task ':distribution:archives:integ-test-zip:integTestCluster#clean', task ':distribution:archives:integ-test-zip:integTestCluster#createCwd', task ':distribution:archives:integ-test-zip:integTestCluster#checkPrevious', task ':distribution:archives:integ-test-zip:integTestCluster#stopPrevious', task ':distribution:archives:integ-test-zip:integTestCluster#extract', task ':distribution:archives:integ-test-zip:integTestCluster#configure', task ':distribution:archives:integ-test-zip:integTestCluster#start', task ':distribution:archives:integ-test-zip:processTestResources', task ':distribution:archives:integ-test-zip:copyRestSpec', task ':distribution:archives:integ-test-zip:copyCheckstyleConf', task ':distribution:archives:integ-test-zip:compileTestJava', task ':distribution:archives:integ-test-zip:testClasses', task ':distribution:archives:integ-test-zip:checkstyleTest', task ':distribution:archives:integ-test-zip:checkstyle', task ':distribution:archives:integ-test-zip:filepermissions', task ':distribution:archives:integ-test-zip:forbiddenApisTest', task ':distribution:archives:integ-test-zip:forbiddenApis', task ':distribution:archives:integ-test-zip:forbiddenPatterns', task ':distribution:archives:integ-test-zip:jarHell', task ':distribution:archives:integ-test-zip:licenseHeaders', task ':distribution:archives:integ-test-zip:loggerUsageCheck', task ':distribution:archives:integ-test-zip:namingConventions', task ':distribution:archives:integ-test-zip:thirdPartyAudit', task ':distribution:archives:integ-test-zip:precommit', task ':distribution:archives:integ-test-zip:integTestCluster#init', task ':distribution:archives:integ-test-zip:integTestCluster#wait', task ':distribution:archives:integ-test-zip:integTestRunner', task ':distribution:archives:integ-test-zip:integTestCluster#stop', task ':distribution:archives:integ-test-zip:integTest', task ':plugins:analysis-icu:integTestCluster#init', task ':plugins:analysis-icu:integTestCluster#wait', task ':plugins:analysis-icu:integTestRunner', task ':plugins:analysis-icu:integTestCluster#stop', task ':plugins:analysis-icu:integTest', task ':plugins:analysis-kuromoji:integTestCluster#prepareCluster.cleanShared', task ':plugins:analysis-kuromoji:integTestCluster#clean', task ':plugins:analysis-kuromoji:integTestCluster#createCwd', task ':plugins:analysis-kuromoji:integTestCluster#checkPrevious', task ':plugins:analysis-kuromoji:integTestCluster#stopPrevious', task ':plugins:analysis-kuromoji:integTestCluster#extract', task ':plugins:analysis-kuromoji:integTestCluster#configure', task ':plugins:analysis-kuromoji:integTestCluster#copyPlugins', task ':plugins:analysis-kuromoji:integTestCluster#installAnalysisKuromojiPlugin', task ':plugins:analysis-kuromoji:integTestCluster#start', task ':plugins:analysis-kuromoji:processTestResources', task ':plugins:analysis-kuromoji:copyRestSpec', task ':plugins:analysis-kuromoji:copyCheckstyleConf', task ':plugins:analysis-kuromoji:checkstyleMain', task ':plugins:analysis-kuromoji:compileTestJava', task ':plugins:analysis-kuromoji:testClasses', task ':plugins:analysis-kuromoji:checkstyleTest', task ':plugins:analysis-kuromoji:checkstyle', task ':plugins:analysis-kuromoji:dependencyLicenses', task ':plugins:analysis-kuromoji:filepermissions', task ':plugins:analysis-kuromoji:forbiddenApisMain', task ':plugins:analysis-kuromoji:forbiddenApisTest', task ':plugins:analysis-kuromoji:forbiddenApis', task ':plugins:analysis-kuromoji:forbiddenPatterns', task ':plugins:analysis-kuromoji:jarHell', task ':plugins:analysis-kuromoji:licenseHeaders', task ':plugins:analysis-kuromoji:loggerUsageCheck', task ':plugins:analysis-kuromoji:namingConventions', task ':plugins:analysis-kuromoji:thirdPartyAudit', task ':plugins:analysis-kuromoji:precommit', task ':plugins:analysis-kuromoji:test', task ':plugins:analysis-kuromoji:integTestCluster#init', task ':plugins:analysis-kuromoji:integTestCluster#wait', task ':plugins:analysis-kuromoji:integTestRunner', task ':plugins:analysis-kuromoji:integTestCluster#stop', task ':plugins:analysis-kuromoji:integTest', task ':plugins:analysis-nori:integTestCluster#prepareCluster.cleanShared', task ':plugins:analysis-nori:integTestCluster#clean', task ':plugins:analysis-nori:integTestCluster#createCwd', task ':plugins:analysis-nori:integTestCluster#checkPrevious', task ':plugins:analysis-nori:integTestCluster#stopPrevious', task ':plugins:analysis-nori:integTestCluster#extract', task ':plugins:analysis-nori:integTestCluster#configure', task ':plugins:analysis-nori:integTestCluster#copyPlugins', task ':plugins:analysis-nori:integTestCluster#installAnalysisNoriPlugin', task ':plugins:analysis-nori:integTestCluster#start', task ':plugins:analysis-nori:processTestResources', task ':plugins:analysis-nori:copyRestSpec', task ':plugins:analysis-nori:copyCheckstyleConf', task ':plugins:analysis-nori:checkstyleMain', task ':plugins:analysis-nori:compileTestJava', task ':plugins:analysis-nori:testClasses', task ':plugins:analysis-nori:checkstyleTest', task ':plugins:analysis-nori:checkstyle', task ':plugins:analysis-nori:dependencyLicenses', task ':plugins:analysis-nori:filepermissions', task ':plugins:analysis-nori:forbiddenApisMain', task ':plugins:analysis-nori:forbiddenApisTest', task ':plugins:analysis-nori:forbiddenApis', task ':plugins:analysis-nori:forbiddenPatterns', task ':plugins:analysis-nori:jarHell', task ':plugins:analysis-nori:licenseHeaders', task ':plugins:analysis-nori:loggerUsageCheck', task ':plugins:analysis-nori:namingConventions', task ':plugins:analysis-nori:thirdPartyAudit', task ':plugins:analysis-nori:precommit', task ':plugins:analysis-nori:test', task ':plugins:analysis-nori:integTestCluster#init', task ':plugins:analysis-nori:integTestCluster#wait', task ':plugins:analysis-nori:integTestRunner', task ':plugins:analysis-nori:integTestCluster#stop', task ':plugins:analysis-nori:integTest', task ':plugins:analysis-phonetic:integTestCluster#prepareCluster.cleanShared', task ':plugins:analysis-phonetic:integTestCluster#clean', task ':plugins:analysis-phonetic:integTestCluster#createCwd', task ':plugins:analysis-phonetic:integTestCluster#checkPrevious', task ':plugins:analysis-phonetic:integTestCluster#stopPrevious', task ':plugins:analysis-phonetic:integTestCluster#extract', task ':plugins:analysis-phonetic:integTestCluster#configure', task ':plugins:analysis-phonetic:integTestCluster#copyPlugins', task ':plugins:analysis-phonetic:integTestCluster#installAnalysisPhoneticPlugin', task ':plugins:analysis-phonetic:integTestCluster#start', task ':plugins:analysis-phonetic:processTestResources', task ':plugins:analysis-phonetic:copyRestSpec', task ':plugins:analysis-phonetic:copyCheckstyleConf', task ':plugins:analysis-phonetic:checkstyleMain', task ':plugins:analysis-phonetic:compileTestJava', task ':plugins:analysis-phonetic:testClasses', task ':plugins:analysis-phonetic:checkstyleTest', task ':plugins:analysis-phonetic:checkstyle', task ':plugins:analysis-phonetic:dependencyLicenses', task ':plugins:analysis-phonetic:filepermissions', task ':plugins:analysis-phonetic:forbiddenApisMain', task ':plugins:analysis-phonetic:forbiddenApisTest', task ':plugins:analysis-phonetic:forbiddenApis', task ':plugins:analysis-phonetic:forbiddenPatterns', task ':plugins:analysis-phonetic:jarHell', task ':plugins:analysis-phonetic:licenseHeaders', task ':plugins:analysis-phonetic:loggerUsageCheck', task ':plugins:analysis-phonetic:namingConventions', task ':plugins:analysis-phonetic:thirdPartyAudit', task ':plugins:analysis-phonetic:precommit', task ':plugins:analysis-phonetic:test', task ':plugins:analysis-phonetic:integTestCluster#init', task ':plugins:analysis-phonetic:integTestCluster#wait', task ':plugins:analysis-phonetic:integTestRunner', task ':plugins:analysis-phonetic:integTestCluster#stop', task ':plugins:analysis-phonetic:integTest', task ':plugins:analysis-smartcn:integTestCluster#prepareCluster.cleanShared', task ':plugins:analysis-smartcn:integTestCluster#clean', task ':plugins:analysis-smartcn:integTestCluster#createCwd', task ':plugins:analysis-smartcn:integTestCluster#checkPrevious', task ':plugins:analysis-smartcn:integTestCluster#stopPrevious', task ':plugins:analysis-smartcn:integTestCluster#extract', task ':plugins:analysis-smartcn:integTestCluster#configure', task ':plugins:analysis-smartcn:integTestCluster#copyPlugins', task ':plugins:analysis-smartcn:integTestCluster#installAnalysisSmartcnPlugin', task ':plugins:analysis-smartcn:integTestCluster#start', task ':plugins:analysis-smartcn:processTestResources', task ':plugins:analysis-smartcn:copyRestSpec', task ':plugins:analysis-smartcn:copyCheckstyleConf', task ':plugins:analysis-smartcn:checkstyleMain', task ':plugins:analysis-smartcn:compileTestJava', task ':plugins:analysis-smartcn:testClasses', task ':plugins:analysis-smartcn:checkstyleTest', task ':plugins:analysis-smartcn:checkstyle', task ':plugins:analysis-smartcn:dependencyLicenses', task ':plugins:analysis-smartcn:filepermissions', task ':plugins:analysis-smartcn:forbiddenApisMain', task ':plugins:analysis-smartcn:forbiddenApisTest', task ':plugins:analysis-smartcn:forbiddenApis', task ':plugins:analysis-smartcn:forbiddenPatterns', task ':plugins:analysis-smartcn:jarHell', task ':plugins:analysis-smartcn:licenseHeaders', task ':plugins:analysis-smartcn:loggerUsageCheck', task ':plugins:analysis-smartcn:namingConventions', task ':plugins:analysis-smartcn:thirdPartyAudit', task ':plugins:analysis-smartcn:precommit', task ':plugins:analysis-smartcn:test', task ':plugins:analysis-smartcn:integTestCluster#init', task ':plugins:analysis-smartcn:integTestCluster#wait', task ':plugins:analysis-smartcn:integTestRunner', task ':plugins:analysis-smartcn:integTestCluster#stop', task ':plugins:analysis-smartcn:integTest', task ':plugins:analysis-stempel:integTestCluster#prepareCluster.cleanShared', task ':plugins:analysis-stempel:integTestCluster#clean', task ':plugins:analysis-stempel:integTestCluster#createCwd', task ':plugins:analysis-stempel:integTestCluster#checkPrevious', task ':plugins:analysis-stempel:integTestCluster#stopPrevious', task ':plugins:analysis-stempel:integTestCluster#extract', task ':plugins:analysis-stempel:integTestCluster#configure', task ':plugins:analysis-stempel:integTestCluster#copyPlugins', task ':plugins:analysis-stempel:integTestCluster#installAnalysisStempelPlugin', task ':plugins:analysis-stempel:integTestCluster#start', task ':plugins:analysis-stempel:processTestResources', task ':plugins:analysis-stempel:copyRestSpec', task ':plugins:analysis-stempel:copyCheckstyleConf', task ':plugins:analysis-stempel:checkstyleMain', task ':plugins:analysis-stempel:compileTestJava', task ':plugins:analysis-stempel:testClasses', task ':plugins:analysis-stempel:checkstyleTest', task ':plugins:analysis-stempel:checkstyle', task ':plugins:analysis-stempel:dependencyLicenses', task ':plugins:analysis-stempel:filepermissions', task ':plugins:analysis-stempel:forbiddenApisMain', task ':plugins:analysis-stempel:forbiddenApisTest', task ':plugins:analysis-stempel:forbiddenApis', task ':plugins:analysis-stempel:forbiddenPatterns', task ':plugins:analysis-stempel:jarHell', task ':plugins:analysis-stempel:licenseHeaders', task ':plugins:analysis-stempel:loggerUsageCheck', task ':plugins:analysis-stempel:namingConventions', task ':plugins:analysis-stempel:thirdPartyAudit', task ':plugins:analysis-stempel:precommit', task ':plugins:analysis-stempel:test', task ':plugins:analysis-stempel:integTestCluster#init', task ':plugins:analysis-stempel:integTestCluster#wait', task ':plugins:analysis-stempel:integTestRunner', task ':plugins:analysis-stempel:integTestCluster#stop', task ':plugins:analysis-stempel:integTest', task ':plugins:analysis-ukrainian:integTestCluster#prepareCluster.cleanShared', task ':plugins:analysis-ukrainian:integTestCluster#clean', task ':plugins:analysis-ukrainian:integTestCluster#createCwd', task ':plugins:analysis-ukrainian:integTestCluster#checkPrevious', task ':plugins:analysis-ukrainian:integTestCluster#stopPrevious', task ':plugins:analysis-ukrainian:integTestCluster#extract', task ':plugins:analysis-ukrainian:integTestCluster#configure', task ':plugins:analysis-ukrainian:integTestCluster#copyPlugins', task ':plugins:analysis-ukrainian:integTestCluster#installAnalysisUkrainianPlugin', task ':plugins:analysis-ukrainian:integTestCluster#start', task ':plugins:analysis-ukrainian:processTestResources', task ':plugins:analysis-ukrainian:copyRestSpec', task ':plugins:analysis-ukrainian:copyCheckstyleConf', task ':plugins:analysis-ukrainian:checkstyleMain', task ':plugins:analysis-ukrainian:compileTestJava', task ':plugins:analysis-ukrainian:testClasses', task ':plugins:analysis-ukrainian:checkstyleTest', task ':plugins:analysis-ukrainian:checkstyle', task ':plugins:analysis-ukrainian:dependencyLicenses', task ':plugins:analysis-ukrainian:filepermissions', task ':plugins:analysis-ukrainian:forbiddenApisMain', task ':plugins:analysis-ukrainian:forbiddenApisTest', task ':plugins:analysis-ukrainian:forbiddenApis', task ':plugins:analysis-ukrainian:forbiddenPatterns', task ':plugins:analysis-ukrainian:jarHell', task ':plugins:analysis-ukrainian:licenseHeaders', task ':plugins:analysis-ukrainian:loggerUsageCheck', task ':plugins:analysis-ukrainian:namingConventions', task ':plugins:analysis-ukrainian:thirdPartyAudit', task ':plugins:analysis-ukrainian:precommit', task ':plugins:analysis-ukrainian:test', task ':plugins:analysis-ukrainian:integTestCluster#init', task ':plugins:analysis-ukrainian:integTestCluster#wait', task ':plugins:analysis-ukrainian:integTestRunner', task ':plugins:analysis-ukrainian:integTestCluster#stop', task ':plugins:analysis-ukrainian:integTest', task ':plugins:discovery-azure-classic:integTestCluster#prepareCluster.cleanShared', task ':plugins:discovery-azure-classic:integTestCluster#clean', task ':plugins:discovery-azure-classic:integTestCluster#createCwd', task ':plugins:discovery-azure-classic:integTestCluster#checkPrevious', task ':plugins:discovery-azure-classic:integTestCluster#stopPrevious', task ':plugins:discovery-azure-classic:integTestCluster#extract', task ':plugins:discovery-azure-classic:integTestCluster#configure', task ':plugins:discovery-azure-classic:integTestCluster#copyPlugins', task ':plugins:discovery-azure-classic:integTestCluster#installDiscoveryAzureClassicPlugin', task ':plugins:discovery-azure-classic:integTestCluster#start', task ':plugins:discovery-azure-classic:createKey', task ':plugins:discovery-azure-classic:processTestResources', task ':plugins:discovery-azure-classic:copyRestSpec', task ':plugins:discovery-azure-classic:copyCheckstyleConf', task ':plugins:discovery-azure-classic:checkstyleMain', task ':plugins:discovery-azure-classic:compileTestJava', task ':plugins:discovery-azure-classic:testClasses', task ':plugins:discovery-azure-classic:checkstyleTest', task ':plugins:discovery-azure-classic:checkstyle', task ':plugins:discovery-azure-classic:dependencyLicenses', task ':plugins:discovery-azure-classic:filepermissions', task ':plugins:discovery-azure-classic:forbiddenApisMain', task ':plugins:discovery-azure-classic:forbiddenApisTest', task ':plugins:discovery-azure-classic:forbiddenApis', task ':plugins:discovery-azure-classic:forbiddenPatterns', task ':plugins:discovery-azure-classic:jarHell', task ':plugins:discovery-azure-classic:licenseHeaders', task ':plugins:discovery-azure-classic:loggerUsageCheck', task ':plugins:discovery-azure-classic:namingConventions', task ':plugins:discovery-azure-classic:thirdPartyAudit', task ':plugins:discovery-azure-classic:precommit', task ':plugins:discovery-azure-classic:test', task ':plugins:discovery-azure-classic:integTestCluster#init', task ':plugins:discovery-azure-classic:integTestCluster#wait', task ':plugins:discovery-azure-classic:integTestRunner', task ':plugins:discovery-azure-classic:integTestCluster#stop', task ':plugins:discovery-azure-classic:integTest', task ':plugins:discovery-ec2:integTestCluster#prepareCluster.cleanShared', task ':plugins:discovery-ec2:integTestCluster#clean', task ':plugins:discovery-ec2:integTestCluster#createCwd', task ':plugins:discovery-ec2:integTestCluster#checkPrevious', task ':plugins:discovery-ec2:integTestCluster#stopPrevious', task ':plugins:discovery-ec2:integTestCluster#extract', task ':plugins:discovery-ec2:integTestCluster#configure', task ':plugins:discovery-ec2:integTestCluster#copyPlugins', task ':plugins:discovery-ec2:integTestCluster#installDiscoveryEc2Plugin', task ':plugins:discovery-ec2:integTestCluster#start', task ':plugins:discovery-ec2:processTestResources', task ':plugins:discovery-ec2:copyRestSpec', task ':plugins:discovery-ec2:copyCheckstyleConf', task ':plugins:discovery-ec2:checkstyleMain', task ':plugins:discovery-ec2:compileTestJava', task ':plugins:discovery-ec2:testClasses', task ':plugins:discovery-ec2:checkstyleTest', task ':plugins:discovery-ec2:checkstyle', task ':plugins:discovery-ec2:dependencyLicenses', task ':plugins:discovery-ec2:filepermissions', task ':plugins:discovery-ec2:forbiddenApisMain', task ':plugins:discovery-ec2:forbiddenApisTest', task ':plugins:discovery-ec2:forbiddenApis', task ':plugins:discovery-ec2:forbiddenPatterns', task ':plugins:discovery-ec2:jarHell', task ':plugins:discovery-ec2:licenseHeaders', task ':plugins:discovery-ec2:loggerUsageCheck', task ':plugins:discovery-ec2:namingConventions', task ':plugins:discovery-ec2:thirdPartyAudit', task ':plugins:discovery-ec2:precommit', task ':plugins:discovery-ec2:test', task ':plugins:discovery-ec2:integTestCluster#init', task ':plugins:discovery-ec2:integTestCluster#wait', task ':plugins:discovery-ec2:integTestRunner', task ':plugins:discovery-ec2:integTestCluster#stop', task ':plugins:discovery-ec2:integTest', task ':plugins:discovery-file:initialCluster#prepareCluster.cleanShared', task ':plugins:discovery-file:initialCluster#clean', task ':plugins:discovery-file:initialCluster#createCwd', task ':plugins:discovery-file:initialCluster#checkPrevious', task ':plugins:discovery-file:initialCluster#stopPrevious', task ':plugins:discovery-file:initialCluster#extract', task ':plugins:discovery-file:initialCluster#configure', task ':plugins:discovery-file:initialCluster#start', task ':plugins:discovery-file:processTestResources', task ':plugins:discovery-file:copyRestSpec', task ':plugins:discovery-file:copyCheckstyleConf', task ':plugins:discovery-file:checkstyleMain', task ':plugins:discovery-file:compileTestJava', task ':plugins:discovery-file:testClasses', task ':plugins:discovery-file:checkstyleTest', task ':plugins:discovery-file:checkstyle', task ':plugins:discovery-file:dependencyLicenses', task ':plugins:discovery-file:filepermissions', task ':plugins:discovery-file:forbiddenApisMain', task ':plugins:discovery-file:forbiddenApisTest', task ':plugins:discovery-file:forbiddenApis', task ':plugins:discovery-file:forbiddenPatterns', task ':plugins:discovery-file:jarHell', task ':plugins:discovery-file:licenseHeaders', task ':plugins:discovery-file:loggerUsageCheck', task ':plugins:discovery-file:namingConventions', task ':plugins:discovery-file:thirdPartyAudit', task ':plugins:discovery-file:precommit', task ':plugins:discovery-file:test', task ':plugins:discovery-file:integTestCluster#init', task ':plugins:discovery-file:initialCluster#wait', task ':plugins:discovery-file:setupSeedNodeAndUnicastHostsFile', task ':plugins:discovery-file:integTestCluster#prepareCluster.cleanShared', task ':plugins:discovery-file:integTestCluster#clean', task ':plugins:discovery-file:integTestCluster#createCwd', task ':plugins:discovery-file:integTestCluster#checkPrevious', task ':plugins:discovery-file:integTestCluster#stopPrevious', task ':plugins:discovery-file:integTestCluster#extract', task ':plugins:discovery-file:integTestCluster#configure', task ':plugins:discovery-file:integTestCluster#copyPlugins', task ':plugins:discovery-file:integTestCluster#installDiscoveryFilePlugin', task ':plugins:discovery-file:integTestCluster#extraConfig', task ':plugins:discovery-file:integTestCluster#start', task ':plugins:discovery-file:integTestCluster#wait', task ':plugins:discovery-file:integTestRunner', task ':plugins:discovery-file:integTestCluster#stop', task ':plugins:discovery-file:initialCluster#stop', task ':plugins:discovery-file:integTest', task ':plugins:discovery-gce:integTestCluster#prepareCluster.cleanShared', task ':plugins:discovery-gce:integTestCluster#clean', task ':plugins:discovery-gce:integTestCluster#createCwd', task ':plugins:discovery-gce:integTestCluster#checkPrevious', task ':plugins:discovery-gce:integTestCluster#stopPrevious', task ':plugins:discovery-gce:integTestCluster#extract', task ':plugins:discovery-gce:integTestCluster#configure', task ':plugins:discovery-gce:integTestCluster#copyPlugins', task ':plugins:discovery-gce:integTestCluster#installDiscoveryGcePlugin', task ':plugins:discovery-gce:integTestCluster#start', task ':plugins:discovery-gce:createKey', task ':plugins:discovery-gce:processTestResources', task ':plugins:discovery-gce:copyRestSpec', task ':plugins:discovery-gce:copyCheckstyleConf', task ':plugins:discovery-gce:checkstyleMain', task ':plugins:discovery-gce:compileTestJava', task ':plugins:discovery-gce:testClasses', task ':plugins:discovery-gce:checkstyleTest', task ':plugins:discovery-gce:checkstyle', task ':plugins:discovery-gce:dependencyLicenses', task ':plugins:discovery-gce:filepermissions', task ':plugins:discovery-gce:forbiddenApisMain', task ':plugins:discovery-gce:forbiddenApisTest', task ':plugins:discovery-gce:forbiddenApis', task ':plugins:discovery-gce:forbiddenPatterns', task ':plugins:discovery-gce:jarHell', task ':plugins:discovery-gce:licenseHeaders', task ':plugins:discovery-gce:loggerUsageCheck', task ':plugins:discovery-gce:namingConventions', task ':plugins:discovery-gce:thirdPartyAudit', task ':plugins:discovery-gce:precommit', task ':plugins:discovery-gce:test', task ':plugins:discovery-gce:integTestCluster#init', task ':plugins:discovery-gce:integTestCluster#wait', task ':plugins:discovery-gce:integTestRunner', task ':plugins:discovery-gce:integTestCluster#stop', task ':plugins:discovery-gce:integTest', task ':plugins:ingest-attachment:integTestCluster#prepareCluster.cleanShared', task ':plugins:ingest-attachment:integTestCluster#clean', task ':plugins:ingest-attachment:integTestCluster#createCwd', task ':plugins:ingest-attachment:integTestCluster#checkPrevious', task ':plugins:ingest-attachment:integTestCluster#stopPrevious', task ':plugins:ingest-attachment:integTestCluster#extract', task ':plugins:ingest-attachment:integTestCluster#configure', task ':plugins:ingest-attachment:integTestCluster#copyPlugins', task ':plugins:ingest-attachment:integTestCluster#installIngestAttachmentPlugin', task ':plugins:ingest-attachment:integTestCluster#start', task ':plugins:ingest-attachment:processTestResources', task ':plugins:ingest-attachment:copyRestSpec', task ':plugins:ingest-attachment:copyCheckstyleConf', task ':plugins:ingest-attachment:checkstyleMain', task ':plugins:ingest-attachment:compileTestJava', task ':plugins:ingest-attachment:testClasses', task ':plugins:ingest-attachment:checkstyleTest', task ':plugins:ingest-attachment:checkstyle', task ':plugins:ingest-attachment:dependencyLicenses', task ':plugins:ingest-attachment:filepermissions', task ':plugins:ingest-attachment:forbiddenApisMain', task ':plugins:ingest-attachment:forbiddenApisTest', task ':plugins:ingest-attachment:forbiddenApis', task ':plugins:ingest-attachment:forbiddenPatterns', task ':plugins:ingest-attachment:jarHell', task ':plugins:ingest-attachment:licenseHeaders', task ':plugins:ingest-attachment:loggerUsageCheck', task ':plugins:ingest-attachment:namingConventions', task ':plugins:ingest-attachment:thirdPartyAudit', task ':plugins:ingest-attachment:precommit', task ':plugins:ingest-attachment:test', task ':plugins:ingest-attachment:integTestCluster#init', task ':plugins:ingest-attachment:integTestCluster#wait', task ':plugins:ingest-attachment:integTestRunner', task ':plugins:ingest-attachment:integTestCluster#stop', task ':plugins:ingest-attachment:integTest', task ':plugins:ingest-geoip:integTestCluster#prepareCluster.cleanShared', task ':plugins:ingest-geoip:integTestCluster#clean', task ':plugins:ingest-geoip:integTestCluster#createCwd', task ':plugins:ingest-geoip:integTestCluster#checkPrevious', task ':plugins:ingest-geoip:integTestCluster#stopPrevious', task ':plugins:ingest-geoip:integTestCluster#extract', task ':plugins:ingest-geoip:integTestCluster#configure', task ':plugins:ingest-geoip:integTestCluster#copyPlugins', task ':plugins:ingest-geoip:integTestCluster#installIngestGeoipPlugin', task ':plugins:ingest-geoip:integTestCluster#start', task ':plugins:ingest-geoip:processTestResources', task ':plugins:ingest-geoip:copyRestSpec', task ':plugins:ingest-geoip:copyCheckstyleConf', task ':plugins:ingest-geoip:checkstyleMain', task ':plugins:ingest-geoip:compileTestJava', task ':plugins:ingest-geoip:testClasses', task ':plugins:ingest-geoip:checkstyleTest', task ':plugins:ingest-geoip:checkstyle', task ':plugins:ingest-geoip:dependencyLicenses', task ':plugins:ingest-geoip:filepermissions', task ':plugins:ingest-geoip:forbiddenApisMain', task ':plugins:ingest-geoip:forbiddenApisTest', task ':plugins:ingest-geoip:forbiddenApis', task ':plugins:ingest-geoip:forbiddenPatterns', task ':plugins:ingest-geoip:jarHell', task ':plugins:ingest-geoip:licenseHeaders', task ':plugins:ingest-geoip:loggerUsageCheck', task ':plugins:ingest-geoip:namingConventions', task ':plugins:ingest-geoip:thirdPartyAudit', task ':plugins:ingest-geoip:precommit', task ':plugins:ingest-geoip:test', task ':plugins:ingest-geoip:integTestCluster#init', task ':plugins:ingest-geoip:integTestCluster#wait', task ':plugins:ingest-geoip:integTestRunner', task ':plugins:ingest-geoip:integTestCluster#stop', task ':plugins:ingest-geoip:integTest', task ':plugins:ingest-user-agent:integTestCluster#prepareCluster.cleanShared', task ':plugins:ingest-user-agent:integTestCluster#clean', task ':plugins:ingest-user-agent:integTestCluster#createCwd', task ':plugins:ingest-user-agent:integTestCluster#checkPrevious', task ':plugins:ingest-user-agent:integTestCluster#stopPrevious', task ':plugins:ingest-user-agent:integTestCluster#extract', task ':plugins:ingest-user-agent:integTestCluster#configure', task ':plugins:ingest-user-agent:integTestCluster#copyPlugins', task ':plugins:ingest-user-agent:integTestCluster#installIngestUserAgentPlugin', task ':plugins:ingest-user-agent:integTestCluster#extraConfig', task ':plugins:ingest-user-agent:integTestCluster#start', task ':plugins:ingest-user-agent:processTestResources', task ':plugins:ingest-user-agent:copyRestSpec', task ':plugins:ingest-user-agent:copyCheckstyleConf', task ':plugins:ingest-user-agent:checkstyleMain', task ':plugins:ingest-user-agent:compileTestJava', task ':plugins:ingest-user-agent:testClasses', task ':plugins:ingest-user-agent:checkstyleTest', task ':plugins:ingest-user-agent:checkstyle', task ':plugins:ingest-user-agent:dependencyLicenses', task ':plugins:ingest-user-agent:filepermissions', task ':plugins:ingest-user-agent:forbiddenApisMain', task ':plugins:ingest-user-agent:forbiddenApisTest', task ':plugins:ingest-user-agent:forbiddenApis', task ':plugins:ingest-user-agent:forbiddenPatterns', task ':plugins:ingest-user-agent:jarHell', task ':plugins:ingest-user-agent:licenseHeaders', task ':plugins:ingest-user-agent:loggerUsageCheck', task ':plugins:ingest-user-agent:namingConventions', task ':plugins:ingest-user-agent:thirdPartyAudit', task ':plugins:ingest-user-agent:precommit', task ':plugins:ingest-user-agent:test', task ':plugins:ingest-user-agent:integTestCluster#init', task ':plugins:ingest-user-agent:integTestCluster#wait', task ':plugins:ingest-user-agent:integTestRunner', task ':plugins:ingest-user-agent:integTestCluster#stop', task ':plugins:ingest-user-agent:integTest', task ':plugins:mapper-murmur3:integTestCluster#prepareCluster.cleanShared', task ':plugins:mapper-murmur3:integTestCluster#clean', task ':plugins:mapper-murmur3:integTestCluster#createCwd', task ':plugins:mapper-murmur3:integTestCluster#checkPrevious', task ':plugins:mapper-murmur3:integTestCluster#stopPrevious', task ':plugins:mapper-murmur3:integTestCluster#extract', task ':plugins:mapper-murmur3:integTestCluster#configure', task ':plugins:mapper-murmur3:integTestCluster#copyPlugins', task ':plugins:mapper-murmur3:integTestCluster#installMapperMurmur3Plugin', task ':plugins:mapper-murmur3:integTestCluster#start', task ':plugins:mapper-murmur3:processTestResources', task ':plugins:mapper-murmur3:copyRestSpec', task ':plugins:mapper-murmur3:copyCheckstyleConf', task ':plugins:mapper-murmur3:checkstyleMain', task ':plugins:mapper-murmur3:compileTestJava', task ':plugins:mapper-murmur3:testClasses', task ':plugins:mapper-murmur3:checkstyleTest', task ':plugins:mapper-murmur3:checkstyle', task ':plugins:mapper-murmur3:dependencyLicenses', task ':plugins:mapper-murmur3:filepermissions', task ':plugins:mapper-murmur3:forbiddenApisMain', task ':plugins:mapper-murmur3:forbiddenApisTest', task ':plugins:mapper-murmur3:forbiddenApis', task ':plugins:mapper-murmur3:forbiddenPatterns', task ':plugins:mapper-murmur3:jarHell', task ':plugins:mapper-murmur3:licenseHeaders', task ':plugins:mapper-murmur3:loggerUsageCheck', task ':plugins:mapper-murmur3:namingConventions', task ':plugins:mapper-murmur3:thirdPartyAudit', task ':plugins:mapper-murmur3:precommit', task ':plugins:mapper-murmur3:test', task ':plugins:mapper-murmur3:integTestCluster#init', task ':plugins:mapper-murmur3:integTestCluster#wait', task ':plugins:mapper-murmur3:integTestRunner', task ':plugins:mapper-murmur3:integTestCluster#stop', task ':plugins:mapper-murmur3:integTest', task ':plugins:mapper-size:integTestCluster#prepareCluster.cleanShared', task ':plugins:mapper-size:integTestCluster#clean', task ':plugins:mapper-size:integTestCluster#createCwd', task ':plugins:mapper-size:integTestCluster#checkPrevious', task ':plugins:mapper-size:integTestCluster#stopPrevious', task ':plugins:mapper-size:integTestCluster#extract', task ':plugins:mapper-size:integTestCluster#configure', task ':plugins:mapper-size:integTestCluster#copyPlugins', task ':plugins:mapper-size:integTestCluster#installMapperSizePlugin', task ':plugins:mapper-size:integTestCluster#start', task ':plugins:mapper-size:processTestResources', task ':plugins:mapper-size:copyRestSpec', task ':plugins:mapper-size:copyCheckstyleConf', task ':plugins:mapper-size:checkstyleMain', task ':plugins:mapper-size:compileTestJava', task ':plugins:mapper-size:testClasses', task ':plugins:mapper-size:checkstyleTest', task ':plugins:mapper-size:checkstyle', task ':plugins:mapper-size:dependencyLicenses', task ':plugins:mapper-size:filepermissions', task ':plugins:mapper-size:forbiddenApisMain', task ':plugins:mapper-size:forbiddenApisTest', task ':plugins:mapper-size:forbiddenApis', task ':plugins:mapper-size:forbiddenPatterns', task ':plugins:mapper-size:jarHell', task ':plugins:mapper-size:licenseHeaders', task ':plugins:mapper-size:loggerUsageCheck', task ':plugins:mapper-size:namingConventions', task ':plugins:mapper-size:thirdPartyAudit', task ':plugins:mapper-size:precommit', task ':plugins:mapper-size:test', task ':plugins:mapper-size:integTestCluster#init', task ':plugins:mapper-size:integTestCluster#wait', task ':plugins:mapper-size:integTestRunner', task ':plugins:mapper-size:integTestCluster#stop', task ':plugins:mapper-size:integTest', task ':plugins:store-smb:integTestCluster#prepareCluster.cleanShared', task ':plugins:store-smb:integTestCluster#clean', task ':plugins:store-smb:integTestCluster#createCwd', task ':plugins:store-smb:integTestCluster#checkPrevious', task ':plugins:store-smb:integTestCluster#stopPrevious', task ':plugins:store-smb:integTestCluster#extract', task ':plugins:store-smb:integTestCluster#configure', task ':plugins:store-smb:integTestCluster#copyPlugins', task ':plugins:store-smb:integTestCluster#installStoreSmbPlugin', task ':plugins:store-smb:integTestCluster#start', task ':plugins:store-smb:processTestResources', task ':plugins:store-smb:copyRestSpec', task ':plugins:store-smb:copyCheckstyleConf', task ':plugins:store-smb:checkstyleMain', task ':plugins:store-smb:compileTestJava', task ':plugins:store-smb:testClasses', task ':plugins:store-smb:checkstyleTest', task ':plugins:store-smb:checkstyle', task ':plugins:store-smb:dependencyLicenses', task ':plugins:store-smb:filepermissions', task ':plugins:store-smb:forbiddenApisMain', task ':plugins:store-smb:forbiddenApisTest', task ':plugins:store-smb:forbiddenApis', task ':plugins:store-smb:forbiddenPatterns', task ':plugins:store-smb:jarHell', task ':plugins:store-smb:licenseHeaders', task ':plugins:store-smb:loggerUsageCheck', task ':plugins:store-smb:namingConventions', task ':plugins:store-smb:thirdPartyAudit', task ':plugins:store-smb:precommit', task ':plugins:store-smb:test', task ':plugins:store-smb:integTestCluster#init', task ':plugins:store-smb:integTestCluster#wait', task ':plugins:store-smb:integTestRunner', task ':plugins:store-smb:integTestCluster#stop', task ':plugins:store-smb:integTest', task ':plugins:transport-nio:integTestCluster#prepareCluster.cleanShared', task ':plugins:transport-nio:integTestCluster#clean', task ':plugins:transport-nio:integTestCluster#createCwd', task ':plugins:transport-nio:integTestCluster#checkPrevious', task ':plugins:transport-nio:integTestCluster#stopPrevious', task ':plugins:transport-nio:integTestCluster#extract', task ':plugins:transport-nio:integTestCluster#configure', task ':plugins:transport-nio:integTestCluster#copyPlugins', task ':plugins:transport-nio:integTestCluster#installTransportNioPlugin', task ':plugins:transport-nio:integTestCluster#start', task ':plugins:transport-nio:processTestResources', task ':plugins:transport-nio:copyRestSpec', task ':plugins:transport-nio:copyCheckstyleConf', task ':plugins:transport-nio:checkstyleMain', task ':plugins:transport-nio:compileTestJava', task ':plugins:transport-nio:testClasses', task ':plugins:transport-nio:checkstyleTest', task ':plugins:transport-nio:checkstyle', task ':plugins:transport-nio:dependencyLicenses', task ':plugins:transport-nio:filepermissions', task ':plugins:transport-nio:forbiddenApisMain', task ':plugins:transport-nio:forbiddenApisTest', task ':plugins:transport-nio:forbiddenApis', task ':plugins:transport-nio:forbiddenPatterns', task ':plugins:transport-nio:jarHell', task ':plugins:transport-nio:licenseHeaders', task ':plugins:transport-nio:loggerUsageCheck', task ':plugins:transport-nio:namingConventions', task ':plugins:transport-nio:thirdPartyAudit', task ':plugins:transport-nio:precommit', task ':libs:elasticsearch-nio:copyCheckstyleConf', task ':libs:elasticsearch-nio:checkstyleMain', task ':libs:elasticsearch-nio:compileTestJava', task ':libs:elasticsearch-nio:processTestResources', task ':libs:elasticsearch-nio:testClasses', task ':libs:elasticsearch-nio:checkstyleTest', task ':libs:elasticsearch-nio:checkstyle', task ':libs:elasticsearch-nio:dependencyLicenses', task ':libs:elasticsearch-nio:filepermissions', task ':libs:elasticsearch-nio:forbiddenApisMain', task ':libs:elasticsearch-nio:forbiddenApisTest', task ':libs:elasticsearch-nio:forbiddenApis', task ':libs:elasticsearch-nio:forbiddenPatterns', task ':libs:elasticsearch-nio:jarHell', task ':libs:elasticsearch-nio:licenseHeaders', task ':libs:elasticsearch-nio:loggerUsageCheck', task ':libs:elasticsearch-nio:namingConventions', task ':libs:elasticsearch-nio:thirdPartyAudit', task ':libs:elasticsearch-nio:precommit', task ':libs:elasticsearch-nio:test', task ':plugins:transport-nio:test', task ':plugins:transport-nio:integTestCluster#init', task ':plugins:transport-nio:integTestCluster#wait', task ':plugins:transport-nio:integTestRunner', task ':plugins:transport-nio:integTestCluster#stop', task ':plugins:transport-nio:integTest', task ':docs:integTestCluster#init', task ':docs:integTestCluster#wait', task ':docs:integTestRunner', task ':docs:integTestCluster#stop', task ':docs:integTest', task ':docs:check', task ':docs:build', task ':rest-api-spec:generatePomFileForNebulaPublication', task ':rest-api-spec:javadoc', task ':rest-api-spec:javadocJar', task ':rest-api-spec:sourcesJar', task ':rest-api-spec:assemble', task ':rest-api-spec:extractJar', task ':rest-api-spec:verifyJarLicense', task ':rest-api-spec:verifyJarNotice', task ':rest-api-spec:extractJavadocJar', task ':rest-api-spec:verifyJavadocJarLicense', task ':rest-api-spec:verifyJavadocJarNotice', task ':rest-api-spec:extractSourcesJar', task ':rest-api-spec:verifySourcesJarLicense', task ':rest-api-spec:verifySourcesJarNotice', task ':rest-api-spec:check', task ':rest-api-spec:build', task ':server:generatePomFileForNebulaPublication', task ':server:javadocJar', task ':server:sourcesJar', task ':server:assemble', task ':server:extractJar', task ':server:verifyJarLicense', task ':server:verifyJarNotice', task ':server:extractJavadocJar', task ':server:verifyJavadocJarLicense', task ':server:verifyJavadocJarNotice', task ':server:extractSourcesJar', task ':server:verifySourcesJarLicense', task ':server:verifySourcesJarNotice', task ':server:check', task ':server:build', task ':client:client-benchmark-noop-api-plugin:compileJava', task ':client:client-benchmark-noop-api-plugin:processResources', task ':client:client-benchmark-noop-api-plugin:classes', task ':client:client-benchmark-noop-api-plugin:jar', task ':client:transport:compileJava', task ':client:transport:processResources', task ':client:transport:classes', task ':client:transport:jar', task ':client:benchmark:compileJava', task ':client:benchmark:processResources', task ':client:benchmark:classes', task ':client:benchmark:copyCheckstyleConf', task ':client:benchmark:checkstyleMain', task ':client:benchmark:compileTestJava', task ':client:benchmark:processTestResources', task ':client:benchmark:testClasses', task ':client:benchmark:checkstyleTest', task ':client:benchmark:forbiddenApisMain', task ':client:benchmark:forbiddenApisTest', task ':client:benchmark:forbiddenApis', task ':client:benchmark:checkstyle', task ':client:benchmark:dependencyLicenses', task ':client:benchmark:filepermissions', task ':client:benchmark:forbiddenPatterns', task ':client:benchmark:jarHell', task ':client:benchmark:licenseHeaders', task ':client:benchmark:loggerUsageCheck', task ':client:benchmark:namingConventions', task ':client:benchmark:thirdPartyAudit', task ':client:benchmark:precommit', task ':client:benchmark:jar', task ':client:benchmark:extractJar', task ':client:benchmark:verifyJarLicense', task ':client:benchmark:verifyJarNotice', task ':client:client-benchmark-noop-api-plugin:javadoc', task ':client:rest:javadoc', task ':modules:lang-mustache:javadoc', task ':modules:parent-join:javadoc', task ':modules:percolator:javadoc', task ':modules:rank-eval:javadoc', task ':modules:reindex:javadoc', task ':modules:transport-netty4:javadoc', task ':client:transport:javadoc', task ':client:benchmark:javadoc', task ':client:benchmark:javadocJar', task ':client:benchmark:extractJavadocJar', task ':client:benchmark:verifyJavadocJarLicense', task ':client:benchmark:verifyJavadocJarNotice', task ':client:benchmark:shadowJar', task ':client:benchmark:extractShadowJar', task ':client:benchmark:verifyShadowJarLicense', task ':client:benchmark:verifyShadowJarNotice', task ':client:benchmark:sourcesJar', task ':client:benchmark:extractSourcesJar', task ':client:benchmark:verifySourcesJarLicense', task ':client:benchmark:verifySourcesJarNotice', task ':client:benchmark:check', task ':client:benchmark:build', task ':client:client-benchmark-noop-api-plugin:copyCheckstyleConf', task ':client:client-benchmark-noop-api-plugin:checkstyleMain', task ':client:client-benchmark-noop-api-plugin:copyPluginPropertiesTemplate', task ':client:client-benchmark-noop-api-plugin:pluginProperties', task ':client:client-benchmark-noop-api-plugin:compileTestJava', task ':client:client-benchmark-noop-api-plugin:processTestResources', task ':client:client-benchmark-noop-api-plugin:testClasses', task ':client:client-benchmark-noop-api-plugin:checkstyleTest', task ':client:client-benchmark-noop-api-plugin:forbiddenApisMain', task ':client:client-benchmark-noop-api-plugin:forbiddenApisTest', task ':client:client-benchmark-noop-api-plugin:forbiddenApis', task ':client:client-benchmark-noop-api-plugin:copyRestSpec', task ':client:client-benchmark-noop-api-plugin:checkstyle', task ':client:client-benchmark-noop-api-plugin:dependencyLicenses', task ':client:client-benchmark-noop-api-plugin:filepermissions', task ':client:client-benchmark-noop-api-plugin:forbiddenPatterns', task ':client:client-benchmark-noop-api-plugin:jarHell', task ':client:client-benchmark-noop-api-plugin:licenseHeaders', task ':client:client-benchmark-noop-api-plugin:loggerUsageCheck', task ':client:client-benchmark-noop-api-plugin:namingConventions', task ':client:client-benchmark-noop-api-plugin:thirdPartyAudit', task ':client:client-benchmark-noop-api-plugin:precommit', task ':client:client-benchmark-noop-api-plugin:test', task ':client:client-benchmark-noop-api-plugin:integTestCluster#init', task ':client:client-benchmark-noop-api-plugin:integTestRunner', task ':client:client-benchmark-noop-api-plugin:integTest', task ':client:client-benchmark-noop-api-plugin:extractJar', task ':client:client-benchmark-noop-api-plugin:verifyJarLicense', task ':client:client-benchmark-noop-api-plugin:verifyJarNotice', task ':client:client-benchmark-noop-api-plugin:javadocJar', task ':client:client-benchmark-noop-api-plugin:extractJavadocJar', task ':client:client-benchmark-noop-api-plugin:verifyJavadocJarLicense', task ':client:client-benchmark-noop-api-plugin:verifyJavadocJarNotice', task ':client:client-benchmark-noop-api-plugin:sourcesJar', task ':client:client-benchmark-noop-api-plugin:extractSourcesJar', task ':client:client-benchmark-noop-api-plugin:verifySourcesJarLicense', task ':client:client-benchmark-noop-api-plugin:verifySourcesJarNotice', task ':client:client-benchmark-noop-api-plugin:check', task ':client:client-benchmark-noop-api-plugin:build', task ':client:rest:generatePomFileForNebulaPublication', task ':client:rest:javadocJar', task ':client:rest:sourcesJar', task ':client:rest:assemble', task ':client:rest:copyCheckstyleConf', task ':client:rest:checkstyleMain', task ':client:test:compileJava', task ':client:test:processResources', task ':client:test:classes', task ':client:test:jar', task ':client:rest:compileTestJava', task ':client:rest:processTestResources', task ':client:rest:testClasses', task ':client:rest:checkstyleTest', task ':client:rest:forbiddenApisMain', task ':client:rest:forbiddenApisTest', task ':client:rest:forbiddenApis', task ':client:rest:checkstyle', task ':client:rest:dependencyLicenses', task ':client:rest:filepermissions', task ':client:rest:forbiddenPatterns', task ':client:rest:jarHell', task ':client:rest:licenseHeaders', task ':client:rest:loggerUsageCheck', task ':client:rest:namingConventions', task ':client:rest:thirdPartyAudit', task ':client:rest:precommit', task ':client:test:copyCheckstyleConf', task ':client:test:checkstyleMain', task ':client:test:compileTestJava', task ':client:test:processTestResources', task ':client:test:testClasses', task ':client:test:checkstyleTest', task ':client:test:checkstyle', task ':client:test:dependencyLicenses', task ':client:test:filepermissions', task ':client:test:forbiddenApisMain', task ':client:test:forbiddenApisTest', task ':client:test:forbiddenApis', task ':client:test:forbiddenPatterns', task ':client:test:jarHell', task ':client:test:licenseHeaders', task ':client:test:loggerUsageCheck', task ':client:test:namingConventions', task ':client:test:thirdPartyAudit', task ':client:test:precommit', task ':client:test:test', task ':client:rest:test', task ':client:rest:extractJar', task ':client:rest:verifyJarLicense', task ':client:rest:verifyJarNotice', task ':client:rest:extractJavadocJar', task ':client:rest:verifyJavadocJarLicense', task ':client:rest:verifyJavadocJarNotice', task ':client:rest:extractSourcesJar', task ':client:rest:verifySourcesJarLicense', task ':client:rest:verifySourcesJarNotice', task ':client:rest:check', task ':client:rest:build', task ':client:rest-high-level:generatePomFileForNebulaPublication', task ':client:rest-high-level:compileJava', task ':client:rest-high-level:processResources', task ':client:rest-high-level:classes', task ':client:rest-high-level:jar', task ':modules:aggs-matrix-stats:javadoc', task ':client:rest-high-level:javadoc', task ':client:rest-high-level:javadocJar', task ':client:rest-high-level:sourcesJar', task ':client:rest-high-level:assemble', task ':client:rest-high-level:copyCheckstyleConf', task ':client:rest-high-level:checkstyleMain', task ':client:rest-high-level:compileTestJava', task ':client:rest-high-level:processTestResources', task ':client:rest-high-level:testClasses', task ':client:rest-high-level:checkstyleTest', task ':client:rest-high-level:forbiddenApisMain', task ':client:rest-high-level:forbiddenApisTest', task ':client:rest-high-level:forbiddenApis', task ':client:rest-high-level:integTestCluster#prepareCluster.cleanShared', task ':client:rest-high-level:integTestCluster#clean', task ':client:rest-high-level:integTestCluster#createCwd', task ':client:rest-high-level:integTestCluster#checkPrevious', task ':client:rest-high-level:integTestCluster#stopPrevious', task ':distribution:buildDefaultModules', task ':distribution:archives:buildZip', task ':client:rest-high-level:integTestCluster#extract', task ':client:rest-high-level:integTestCluster#configure', task ':client:rest-high-level:integTestCluster#start', task ':client:rest-high-level:copyRestSpec', task ':client:rest-high-level:checkstyle', task ':client:rest-high-level:dependencyLicenses', task ':client:rest-high-level:filepermissions', task ':client:rest-high-level:forbiddenPatterns', task ':client:rest-high-level:jarHell', task ':client:rest-high-level:licenseHeaders', task ':client:rest-high-level:loggerUsageCheck', task ':client:rest-high-level:namingConventions', task ':client:rest-high-level:thirdPartyAudit', task ':client:rest-high-level:precommit', task ':modules:aggs-matrix-stats:integTestCluster#prepareCluster.cleanShared', task ':modules:aggs-matrix-stats:integTestCluster#clean', task ':modules:aggs-matrix-stats:integTestCluster#createCwd', task ':modules:aggs-matrix-stats:integTestCluster#checkPrevious', task ':modules:aggs-matrix-stats:integTestCluster#stopPrevious', task ':modules:aggs-matrix-stats:integTestCluster#extract', task ':modules:aggs-matrix-stats:integTestCluster#configure', task ':modules:aggs-matrix-stats:integTestCluster#installAggsMatrixStatsModule', task ':modules:aggs-matrix-stats:integTestCluster#start', task ':modules:aggs-matrix-stats:processTestResources', task ':modules:aggs-matrix-stats:copyRestSpec', task ':modules:aggs-matrix-stats:copyCheckstyleConf', task ':modules:aggs-matrix-stats:checkstyleMain', task ':modules:aggs-matrix-stats:compileTestJava', task ':modules:aggs-matrix-stats:testClasses', task ':modules:aggs-matrix-stats:checkstyleTest', task ':modules:aggs-matrix-stats:checkstyle', task ':modules:aggs-matrix-stats:dependencyLicenses', task ':modules:aggs-matrix-stats:filepermissions', task ':modules:aggs-matrix-stats:forbiddenApisMain', task ':modules:aggs-matrix-stats:forbiddenApisTest', task ':modules:aggs-matrix-stats:forbiddenApis', task ':modules:aggs-matrix-stats:forbiddenPatterns', task ':modules:aggs-matrix-stats:jarHell', task ':modules:aggs-matrix-stats:licenseHeaders', task ':modules:aggs-matrix-stats:loggerUsageCheck', task ':modules:aggs-matrix-stats:namingConventions', task ':modules:aggs-matrix-stats:thirdPartyAudit', task ':modules:aggs-matrix-stats:precommit', task ':modules:aggs-matrix-stats:test', task ':modules:aggs-matrix-stats:integTestCluster#init', task ':modules:aggs-matrix-stats:integTestCluster#wait', task ':modules:aggs-matrix-stats:integTestRunner', task ':modules:aggs-matrix-stats:integTestCluster#stop', task ':modules:aggs-matrix-stats:integTest', task ':modules:parent-join:integTestCluster#prepareCluster.cleanShared', task ':modules:parent-join:integTestCluster#clean', task ':modules:parent-join:integTestCluster#createCwd', task ':modules:parent-join:integTestCluster#checkPrevious', task ':modules:parent-join:integTestCluster#stopPrevious', task ':modules:parent-join:integTestCluster#extract', task ':modules:parent-join:integTestCluster#configure', task ':modules:parent-join:integTestCluster#installParentJoinModule', task ':modules:parent-join:integTestCluster#start', task ':modules:parent-join:processTestResources', task ':modules:parent-join:copyRestSpec', task ':modules:parent-join:copyCheckstyleConf', task ':modules:parent-join:checkstyleMain', task ':modules:parent-join:compileTestJava', task ':modules:parent-join:testClasses', task ':modules:parent-join:checkstyleTest', task ':modules:parent-join:checkstyle', task ':modules:parent-join:dependencyLicenses', task ':modules:parent-join:filepermissions', task ':modules:parent-join:forbiddenApisMain', task ':modules:parent-join:forbiddenApisTest', task ':modules:parent-join:forbiddenApis', task ':modules:parent-join:forbiddenPatterns', task ':modules:parent-join:jarHell', task ':modules:parent-join:licenseHeaders', task ':modules:parent-join:loggerUsageCheck', task ':modules:parent-join:namingConventions', task ':modules:parent-join:thirdPartyAudit', task ':modules:parent-join:precommit', task ':modules:parent-join:test', task ':modules:parent-join:integTestCluster#init', task ':modules:parent-join:integTestCluster#wait', task ':modules:parent-join:integTestRunner', task ':modules:parent-join:integTestCluster#stop', task ':modules:parent-join:integTest', task ':modules:rank-eval:integTestCluster#prepareCluster.cleanShared', task ':modules:rank-eval:integTestCluster#clean', task ':modules:rank-eval:integTestCluster#createCwd', task ':modules:rank-eval:integTestCluster#checkPrevious', task ':modules:rank-eval:integTestCluster#stopPrevious', task ':modules:rank-eval:integTestCluster#extract', task ':modules:rank-eval:integTestCluster#configure', task ':modules:rank-eval:integTestCluster#installRankEvalModule', task ':modules:rank-eval:integTestCluster#start', task ':modules:rank-eval:processTestResources', task ':modules:rank-eval:copyRestSpec', task ':modules:rank-eval:copyCheckstyleConf', task ':modules:rank-eval:checkstyleMain', task ':modules:rank-eval:compileTestJava', task ':modules:rank-eval:testClasses', task ':modules:rank-eval:checkstyleTest', task ':modules:rank-eval:checkstyle', task ':modules:rank-eval:dependencyLicenses', task ':modules:rank-eval:filepermissions', task ':modules:rank-eval:forbiddenApisMain', task ':modules:rank-eval:forbiddenApisTest', task ':modules:rank-eval:forbiddenApis', task ':modules:rank-eval:forbiddenPatterns', task ':modules:rank-eval:jarHell', task ':modules:rank-eval:licenseHeaders', task ':modules:rank-eval:loggerUsageCheck', task ':modules:rank-eval:namingConventions', task ':modules:rank-eval:thirdPartyAudit', task ':modules:rank-eval:precommit', task ':modules:rank-eval:test', task ':modules:rank-eval:integTestCluster#init', task ':modules:rank-eval:integTestCluster#wait', task ':modules:rank-eval:integTestRunner', task ':modules:rank-eval:integTestCluster#stop', task ':modules:rank-eval:integTest', task ':client:rest-high-level:integTestCluster#init', task ':client:rest-high-level:integTestCluster#wait', task ':client:rest-high-level:integTestRunner', task ':client:rest-high-level:integTestCluster#stop', task ':client:rest-high-level:integTest', task ':client:rest-high-level:test', task ':client:rest-high-level:extractJar', task ':client:rest-high-level:verifyJarLicense', task ':client:rest-high-level:verifyJarNotice', task ':client:rest-high-level:extractJavadocJar', task ':client:rest-high-level:verifyJavadocJarLicense', task ':client:rest-high-level:verifyJavadocJarNotice', task ':client:rest-high-level:extractSourcesJar', task ':client:rest-high-level:verifySourcesJarLicense', task ':client:rest-high-level:verifySourcesJarNotice', task ':client:rest-high-level:check', task ':client:rest-high-level:build', task ':client:sniffer:generatePomFileForNebulaPublication', task ':client:sniffer:javadoc', task ':client:sniffer:javadocJar', task ':client:sniffer:sourcesJar', task ':client:sniffer:assemble', task ':client:sniffer:copyCheckstyleConf', task ':client:sniffer:checkstyleMain', task ':client:sniffer:compileTestJava', task ':client:sniffer:processTestResources', task ':client:sniffer:testClasses', task ':client:sniffer:checkstyleTest', task ':client:sniffer:forbiddenApisMain', task ':client:sniffer:forbiddenApisTest', task ':client:sniffer:forbiddenApis', task ':client:sniffer:checkstyle', task ':client:sniffer:dependencyLicenses', task ':client:sniffer:filepermissions', task ':client:sniffer:forbiddenPatterns', task ':client:sniffer:jarHell', task ':client:sniffer:licenseHeaders', task ':client:sniffer:loggerUsageCheck', task ':client:sniffer:namingConventions', task ':client:sniffer:thirdPartyAudit', task ':client:sniffer:precommit', task ':client:sniffer:test', task ':client:sniffer:extractJar', task ':client:sniffer:verifyJarLicense', task ':client:sniffer:verifyJarNotice', task ':client:sniffer:extractJavadocJar', task ':client:sniffer:verifyJavadocJarLicense', task ':client:sniffer:verifyJavadocJarNotice', task ':client:sniffer:extractSourcesJar', task ':client:sniffer:verifySourcesJarLicense', task ':client:sniffer:verifySourcesJarNotice', task ':client:sniffer:check', task ':client:sniffer:build', task ':client:test:javadoc', task ':client:test:javadocJar', task ':client:test:sourcesJar', task ':client:test:assemble', task ':client:test:animalsnifferMain', task ':client:test:animalsnifferTest', task ':client:test:extractJar', task ':client:test:verifyJarLicense', task ':client:test:verifyJarNotice', task ':client:test:extractJavadocJar', task ':client:test:verifyJavadocJarLicense', task ':client:test:verifyJavadocJarNotice', task ':client:test:extractSourcesJar', task ':client:test:verifySourcesJarLicense', task ':client:test:verifySourcesJarNotice', task ':client:test:check', task ':client:test:build', task ':client:transport:generatePomFileForNebulaPublication', task ':client:transport:javadocJar', task ':client:transport:sourcesJar', task ':client:transport:assemble', task ':client:transport:copyCheckstyleConf', task ':client:transport:checkstyleMain', task ':client:transport:compileTestJava', task ':client:transport:processTestResources', task ':client:transport:testClasses', task ':client:transport:checkstyleTest', task ':client:transport:forbiddenApisMain', task ':client:transport:forbiddenApisTest', task ':client:transport:forbiddenApis', task ':client:transport:checkstyle', task ':client:transport:dependencyLicenses', task ':client:transport:filepermissions', task ':client:transport:forbiddenPatterns', task ':client:transport:jarHell', task ':client:transport:licenseHeaders', task ':client:transport:loggerUsageCheck', task ':client:transport:namingConventions', task ':client:transport:thirdPartyAudit', task ':client:transport:precommit', task ':modules:lang-mustache:copyCheckstyleConf', task ':modules:lang-mustache:checkstyleMain', task ':modules:lang-mustache:compileTestJava', task ':modules:lang-mustache:processTestResources', task ':modules:lang-mustache:testClasses', task ':modules:lang-mustache:checkstyleTest', task ':modules:lang-mustache:checkstyle', task ':modules:lang-mustache:dependencyLicenses', task ':modules:lang-mustache:filepermissions', task ':modules:lang-mustache:forbiddenApisMain', task ':modules:lang-mustache:forbiddenApisTest', task ':modules:lang-mustache:forbiddenApis', task ':modules:lang-mustache:forbiddenPatterns', task ':modules:lang-mustache:jarHell', task ':modules:lang-mustache:licenseHeaders', task ':modules:lang-mustache:loggerUsageCheck', task ':modules:lang-mustache:namingConventions', task ':modules:lang-mustache:thirdPartyAudit', task ':modules:lang-mustache:precommit', task ':modules:lang-mustache:test', task ':modules:percolator:copyCheckstyleConf', task ':modules:percolator:checkstyleMain', task ':modules:percolator:compileTestJava', task ':modules:percolator:processTestResources', task ':modules:percolator:testClasses', task ':modules:percolator:checkstyleTest', task ':modules:percolator:checkstyle', task ':modules:percolator:dependencyLicenses', task ':modules:percolator:filepermissions', task ':modules:percolator:forbiddenApisMain', task ':modules:percolator:forbiddenApisTest', task ':modules:percolator:forbiddenApis', task ':modules:percolator:forbiddenPatterns', task ':modules:percolator:jarHell', task ':modules:percolator:licenseHeaders', task ':modules:percolator:loggerUsageCheck', task ':modules:percolator:namingConventions', task ':modules:percolator:thirdPartyAudit', task ':modules:percolator:precommit', task ':modules:percolator:test', task ':modules:reindex:copyCheckstyleConf', task ':modules:reindex:checkstyleMain', task ':modules:reindex:copyParentJoinMetadata', task ':modules:reindex:copyTransportNetty4Metadata', task ':modules:reindex:compileTestJava', task ':modules:reindex:processTestResources', task ':modules:reindex:testClasses', task ':modules:reindex:checkstyleTest', task ':modules:reindex:checkstyle', task ':modules:reindex:dependencyLicenses', task ':modules:reindex:filepermissions', task ':modules:reindex:forbiddenApisMain', task ':modules:reindex:forbiddenApisTest', task ':modules:reindex:forbiddenApis', task ':modules:reindex:forbiddenPatterns', task ':modules:reindex:jarHell', task ':modules:reindex:licenseHeaders', task ':modules:reindex:loggerUsageCheck', task ':modules:reindex:namingConventions', task ':modules:reindex:thirdPartyAudit', task ':modules:reindex:precommit', task ':modules:transport-netty4:copyCheckstyleConf', task ':modules:transport-netty4:checkstyleMain', task ':modules:transport-netty4:compileTestJava', task ':modules:transport-netty4:processTestResources', task ':modules:transport-netty4:testClasses', task ':modules:transport-netty4:checkstyleTest', task ':modules:transport-netty4:checkstyle', task ':modules:transport-netty4:dependencyLicenses', task ':modules:transport-netty4:filepermissions', task ':modules:transport-netty4:forbiddenApisMain', task ':modules:transport-netty4:forbiddenApisTest', task ':modules:transport-netty4:forbiddenApis', task ':modules:transport-netty4:forbiddenPatterns', task ':modules:transport-netty4:jarHell', task ':modules:transport-netty4:licenseHeaders', task ':modules:transport-netty4:loggerUsageCheck', task ':modules:transport-netty4:namingConventions', task ':modules:transport-netty4:thirdPartyAudit', task ':modules:transport-netty4:precommit', task ':modules:transport-netty4:test', task ':test:fixtures:old-elasticsearch:compileJava', task ':test:fixtures:old-elasticsearch:processResources', task ':test:fixtures:old-elasticsearch:classes', task ':test:fixtures:old-elasticsearch:copyCheckstyleConf', task ':test:fixtures:old-elasticsearch:checkstyleMain', task ':test:fixtures:old-elasticsearch:compileTestJava', task ':test:fixtures:old-elasticsearch:processTestResources', task ':test:fixtures:old-elasticsearch:testClasses', task ':test:fixtures:old-elasticsearch:checkstyleTest', task ':test:fixtures:old-elasticsearch:checkstyle', task ':test:fixtures:old-elasticsearch:dependencyLicenses', task ':test:fixtures:old-elasticsearch:filepermissions', task ':test:fixtures:old-elasticsearch:forbiddenApisMain', task ':test:fixtures:old-elasticsearch:forbiddenApisTest', task ':test:fixtures:old-elasticsearch:forbiddenApis', task ':test:fixtures:old-elasticsearch:forbiddenPatterns', task ':test:fixtures:old-elasticsearch:jarHell', task ':test:fixtures:old-elasticsearch:licenseHeaders', task ':test:fixtures:old-elasticsearch:loggerUsageCheck', task ':test:fixtures:old-elasticsearch:namingConventions', task ':test:fixtures:old-elasticsearch:thirdPartyAudit', task ':test:fixtures:old-elasticsearch:precommit', task ':test:fixtures:old-elasticsearch:test', task ':modules:reindex:test', task ':client:transport:test', task ':client:transport:extractJar', task ':client:transport:verifyJarLicense', task ':client:transport:verifyJarNotice', task ':client:transport:extractJavadocJar', task ':client:transport:verifyJavadocJarLicense', task ':client:transport:verifyJavadocJarNotice', task ':client:transport:extractSourcesJar', task ':client:transport:verifySourcesJarLicense', task ':client:transport:verifySourcesJarNotice', task ':client:transport:check', task ':client:transport:build', task ':distribution:archives:assemble', task ':distribution:archives:check', task ':distribution:archives:build', task ':distribution:packages:assemble', task ':distribution:packages:check', task ':distribution:packages:build', task ':example-plugins:custom-settings:compileJava', task ':example-plugins:custom-settings:processResources', task ':example-plugins:custom-settings:classes', task ':example-plugins:custom-settings:copyCheckstyleConf', task ':example-plugins:custom-settings:checkstyleMain', task ':example-plugins:custom-settings:copyPluginPropertiesTemplate', task ':example-plugins:custom-settings:pluginProperties', task ':example-plugins:custom-settings:compileTestJava', task ':example-plugins:custom-settings:processTestResources', task ':example-plugins:custom-settings:testClasses', task ':example-plugins:custom-settings:checkstyleTest', task ':example-plugins:custom-settings:forbiddenApisMain', task ':example-plugins:custom-settings:forbiddenApisTest', task ':example-plugins:custom-settings:forbiddenApis', task ':example-plugins:custom-settings:jar', task ':example-plugins:custom-settings:bundlePlugin', task ':example-plugins:custom-settings:integTestCluster#prepareCluster.cleanShared', task ':example-plugins:custom-settings:integTestCluster#clean', task ':example-plugins:custom-settings:integTestCluster#createCwd', task ':example-plugins:custom-settings:integTestCluster#checkPrevious', task ':example-plugins:custom-settings:integTestCluster#stopPrevious', task ':example-plugins:custom-settings:integTestCluster#extract', task ':example-plugins:custom-settings:integTestCluster#configure', task ':example-plugins:custom-settings:integTestCluster#createKeystore', task ':example-plugins:custom-settings:integTestCluster#addToKeystore#custom.secured', task ':example-plugins:custom-settings:integTestCluster#copyPlugins', task ':example-plugins:custom-settings:integTestCluster#installCustomSettingsPlugin', task ':example-plugins:custom-settings:integTestCluster#start', task ':example-plugins:custom-settings:copyRestSpec', task ':example-plugins:custom-settings:checkstyle', task ':example-plugins:custom-settings:dependencyLicenses', task ':example-plugins:custom-settings:filepermissions', task ':example-plugins:custom-settings:forbiddenPatterns', task ':example-plugins:custom-settings:jarHell', task ':example-plugins:custom-settings:licenseHeaders', task ':example-plugins:custom-settings:loggerUsageCheck', task ':example-plugins:custom-settings:namingConventions', task ':example-plugins:custom-settings:thirdPartyAudit', task ':example-plugins:custom-settings:precommit', task ':example-plugins:custom-settings:test', task ':example-plugins:custom-settings:integTestCluster#init', task ':example-plugins:custom-settings:integTestCluster#wait', task ':example-plugins:custom-settings:integTestRunner', task ':example-plugins:custom-settings:integTestCluster#stop', task ':example-plugins:custom-settings:integTest', task ':example-plugins:custom-settings:extractJar', task ':example-plugins:custom-settings:verifyJarLicense', task ':example-plugins:custom-settings:verifyJarNotice', task ':example-plugins:custom-settings:javadoc', task ':example-plugins:custom-settings:javadocJar', task ':example-plugins:custom-settings:extractJavadocJar', task ':example-plugins:custom-settings:verifyJavadocJarLicense', task ':example-plugins:custom-settings:verifyJavadocJarNotice', task ':example-plugins:custom-settings:sourcesJar', task ':example-plugins:custom-settings:extractSourcesJar', task ':example-plugins:custom-settings:verifySourcesJarLicense', task ':example-plugins:custom-settings:verifySourcesJarNotice', task ':example-plugins:custom-settings:check', task ':example-plugins:custom-settings:build', task ':example-plugins:meta-plugin:copyCheckstyleConf', task ':example-plugins:meta-plugin:compileTestJava', task ':example-plugins:meta-plugin:processTestResources', task ':example-plugins:meta-plugin:testClasses', task ':example-plugins:meta-plugin:checkstyleTest', task ':example-plugins:meta-plugin:forbiddenApisTest', task ':example-plugins:meta-plugin:forbiddenApis', task ':example-plugins:meta-plugin:copyPluginPropertiesTemplate', task ':example-plugins:meta-plugin:pluginProperties', task ':example-plugins:meta-plugin:dummy-plugin1:compileJava', task ':example-plugins:meta-plugin:dummy-plugin1:processResources', task ':example-plugins:meta-plugin:dummy-plugin1:classes', task ':example-plugins:meta-plugin:dummy-plugin1:jar', task ':example-plugins:meta-plugin:dummy-plugin1:copyPluginPropertiesTemplate', task ':example-plugins:meta-plugin:dummy-plugin1:pluginProperties', task ':example-plugins:meta-plugin:dummy-plugin1:bundlePlugin', task ':example-plugins:meta-plugin:dummy-plugin2:compileJava', task ':example-plugins:meta-plugin:dummy-plugin2:processResources', task ':example-plugins:meta-plugin:dummy-plugin2:classes', task ':example-plugins:meta-plugin:dummy-plugin2:jar', task ':example-plugins:meta-plugin:dummy-plugin2:copyPluginPropertiesTemplate', task ':example-plugins:meta-plugin:dummy-plugin2:pluginProperties', task ':example-plugins:meta-plugin:dummy-plugin2:bundlePlugin', task ':example-plugins:meta-plugin:bundlePlugin', task ':example-plugins:meta-plugin:integTestCluster#prepareCluster.cleanShared', task ':example-plugins:meta-plugin:integTestCluster#clean', task ':example-plugins:meta-plugin:integTestCluster#createCwd', task ':example-plugins:meta-plugin:integTestCluster#checkPrevious', task ':example-plugins:meta-plugin:integTestCluster#stopPrevious', task ':example-plugins:meta-plugin:integTestCluster#extract', task ':example-plugins:meta-plugin:integTestCluster#configure', task ':example-plugins:meta-plugin:integTestCluster#copyPlugins', task ':example-plugins:meta-plugin:integTestCluster#installMetaPluginPlugin', task ':example-plugins:meta-plugin:integTestCluster#start', task ':example-plugins:meta-plugin:copyRestSpec', task ':example-plugins:meta-plugin:checkstyle', task ':example-plugins:meta-plugin:filepermissions', task ':example-plugins:meta-plugin:forbiddenPatterns', task ':example-plugins:meta-plugin:jarHell', task ':example-plugins:meta-plugin:licenseHeaders', task ':example-plugins:meta-plugin:loggerUsageCheck', task ':example-plugins:meta-plugin:namingConventions', task ':example-plugins:meta-plugin:thirdPartyAudit', task ':example-plugins:meta-plugin:precommit', task ':example-plugins:meta-plugin:integTestCluster#init', task ':example-plugins:meta-plugin:integTestCluster#wait', task ':example-plugins:meta-plugin:integTestRunner', task ':example-plugins:meta-plugin:integTestCluster#stop', task ':example-plugins:meta-plugin:integTest', task ':example-plugins:meta-plugin:check', task ':example-plugins:meta-plugin:build', task ':example-plugins:painless-whitelist:compileJava', task ':example-plugins:painless-whitelist:processResources', task ':example-plugins:painless-whitelist:classes', task ':example-plugins:painless-whitelist:copyCheckstyleConf', task ':example-plugins:painless-whitelist:checkstyleMain', task ':example-plugins:painless-whitelist:copyPluginPropertiesTemplate', task ':example-plugins:painless-whitelist:pluginProperties', task ':example-plugins:painless-whitelist:compileTestJava', task ':example-plugins:painless-whitelist:processTestResources', task ':example-plugins:painless-whitelist:testClasses', task ':example-plugins:painless-whitelist:checkstyleTest', task ':example-plugins:painless-whitelist:forbiddenApisMain', task ':example-plugins:painless-whitelist:forbiddenApisTest', task ':example-plugins:painless-whitelist:forbiddenApis', task ':example-plugins:painless-whitelist:jar', task ':example-plugins:painless-whitelist:bundlePlugin', task ':example-plugins:painless-whitelist:integTestCluster#prepareCluster.cleanShared', task ':example-plugins:painless-whitelist:integTestCluster#clean', task ':example-plugins:painless-whitelist:integTestCluster#createCwd', task ':example-plugins:painless-whitelist:integTestCluster#checkPrevious', task ':example-plugins:painless-whitelist:integTestCluster#stopPrevious', task ':example-plugins:painless-whitelist:integTestCluster#extract', task ':example-plugins:painless-whitelist:integTestCluster#configure', task ':example-plugins:painless-whitelist:integTestCluster#copyPlugins', task ':example-plugins:painless-whitelist:integTestCluster#installPainlessWhitelistPlugin', task ':example-plugins:painless-whitelist:integTestCluster#start', task ':example-plugins:painless-whitelist:copyRestSpec', task ':example-plugins:painless-whitelist:checkstyle', task ':example-plugins:painless-whitelist:dependencyLicenses', task ':example-plugins:painless-whitelist:filepermissions', task ':example-plugins:painless-whitelist:forbiddenPatterns', task ':example-plugins:painless-whitelist:jarHell', task ':example-plugins:painless-whitelist:licenseHeaders', task ':example-plugins:painless-whitelist:loggerUsageCheck', task ':example-plugins:painless-whitelist:namingConventions', task ':example-plugins:painless-whitelist:thirdPartyAudit', task ':example-plugins:painless-whitelist:precommit', task ':modules:lang-painless:copyCheckstyleConf', task ':modules:lang-painless:checkstyleMain', task ':modules:lang-painless:compileTestJava', task ':modules:lang-painless:processTestResources', task ':modules:lang-painless:testClasses', task ':modules:lang-painless:checkstyleTest', task ':modules:lang-painless:checkstyle', task ':modules:lang-painless:dependencyLicenses', task ':modules:lang-painless:filepermissions', task ':modules:lang-painless:forbiddenApisMain', task ':modules:lang-painless:forbiddenApisTest', task ':modules:lang-painless:forbiddenApis', task ':modules:lang-painless:forbiddenPatterns', task ':modules:lang-painless:jarHell', task ':modules:lang-painless:licenseHeaders', task ':modules:lang-painless:loggerUsageCheck', task ':modules:lang-painless:namingConventions', task ':modules:lang-painless:thirdPartyAudit', task ':modules:lang-painless:precommit', task ':modules:lang-painless:spi:copyCheckstyleConf', task ':modules:lang-painless:spi:checkstyleMain', task ':modules:lang-painless:spi:compileTestJava', task ':modules:lang-painless:spi:processTestResources', task ':modules:lang-painless:spi:testClasses', task ':modules:lang-painless:spi:checkstyleTest', task ':modules:lang-painless:spi:checkstyle', task ':modules:lang-painless:spi:dependencyLicenses', task ':modules:lang-painless:spi:filepermissions', task ':modules:lang-painless:spi:forbiddenApisMain', task ':modules:lang-painless:spi:forbiddenApisTest', task ':modules:lang-painless:spi:forbiddenApis', task ':modules:lang-painless:spi:forbiddenPatterns', task ':modules:lang-painless:spi:jarHell', task ':modules:lang-painless:spi:licenseHeaders', task ':modules:lang-painless:spi:loggerUsageCheck', task ':modules:lang-painless:spi:namingConventions', task ':modules:lang-painless:spi:thirdPartyAudit', task ':modules:lang-painless:spi:precommit', task ':modules:lang-painless:spi:test', task ':modules:lang-painless:test', task ':example-plugins:painless-whitelist:test', task ':modules:lang-painless:integTestCluster#prepareCluster.cleanShared', task ':modules:lang-painless:integTestCluster#clean', task ':modules:lang-painless:integTestCluster#createCwd', task ':modules:lang-painless:integTestCluster#checkPrevious', task ':modules:lang-painless:integTestCluster#stopPrevious', task ':modules:lang-painless:integTestCluster#extract', task ':modules:lang-painless:integTestCluster#configure', task ':modules:lang-painless:integTestCluster#installMapperExtrasModule', task ':modules:lang-painless:integTestCluster#installLangPainlessModule', task ':modules:lang-painless:integTestCluster#start', task ':modules:lang-painless:copyRestSpec', task ':modules:lang-painless:integTestCluster#init', task ':modules:lang-painless:integTestCluster#wait', task ':modules:lang-painless:integTestRunner', task ':modules:lang-painless:integTestCluster#stop', task ':modules:lang-painless:integTest', task ':example-plugins:painless-whitelist:integTestCluster#init', task ':example-plugins:painless-whitelist:integTestCluster#wait', task ':example-plugins:painless-whitelist:integTestRunner', task ':example-plugins:painless-whitelist:integTestCluster#stop', task ':example-plugins:painless-whitelist:integTest', task ':example-plugins:painless-whitelist:extractJar', task ':example-plugins:painless-whitelist:verifyJarLicense', task ':example-plugins:painless-whitelist:verifyJarNotice', task ':example-plugins:painless-whitelist:javadoc', task ':example-plugins:painless-whitelist:javadocJar', task ':example-plugins:painless-whitelist:extractJavadocJar', task ':example-plugins:painless-whitelist:verifyJavadocJarLicense', task ':example-plugins:painless-whitelist:verifyJavadocJarNotice', task ':example-plugins:painless-whitelist:sourcesJar', task ':example-plugins:painless-whitelist:extractSourcesJar', task ':example-plugins:painless-whitelist:verifySourcesJarLicense', task ':example-plugins:painless-whitelist:verifySourcesJarNotice', task ':example-plugins:painless-whitelist:check', task ':example-plugins:painless-whitelist:build', task ':example-plugins:rescore:compileJava', task ':example-plugins:rescore:processResources', task ':example-plugins:rescore:classes', task ':example-plugins:rescore:copyCheckstyleConf', task ':example-plugins:rescore:checkstyleMain', task ':example-plugins:rescore:copyPluginPropertiesTemplate', task ':example-plugins:rescore:pluginProperties', task ':example-plugins:rescore:compileTestJava', task ':example-plugins:rescore:processTestResources', task ':example-plugins:rescore:testClasses', task ':example-plugins:rescore:checkstyleTest', task ':example-plugins:rescore:forbiddenApisMain', task ':example-plugins:rescore:forbiddenApisTest', task ':example-plugins:rescore:forbiddenApis', task ':example-plugins:rescore:jar', task ':example-plugins:rescore:bundlePlugin', task ':example-plugins:rescore:integTestCluster#prepareCluster.cleanShared', task ':example-plugins:rescore:integTestCluster#clean', task ':example-plugins:rescore:integTestCluster#createCwd', task ':example-plugins:rescore:integTestCluster#checkPrevious', task ':example-plugins:rescore:integTestCluster#stopPrevious', task ':example-plugins:rescore:integTestCluster#extract', task ':example-plugins:rescore:integTestCluster#configure', task ':example-plugins:rescore:integTestCluster#copyPlugins', task ':example-plugins:rescore:integTestCluster#installRescorePlugin', task ':example-plugins:rescore:integTestCluster#start', task ':example-plugins:rescore:copyRestSpec', task ':example-plugins:rescore:checkstyle', task ':example-plugins:rescore:dependencyLicenses', task ':example-plugins:rescore:filepermissions', task ':example-plugins:rescore:forbiddenPatterns', task ':example-plugins:rescore:jarHell', task ':example-plugins:rescore:licenseHeaders', task ':example-plugins:rescore:loggerUsageCheck', task ':example-plugins:rescore:namingConventions', task ':example-plugins:rescore:thirdPartyAudit', task ':example-plugins:rescore:precommit', task ':example-plugins:rescore:test', task ':example-plugins:rescore:integTestCluster#init', task ':example-plugins:rescore:integTestCluster#wait', task ':example-plugins:rescore:integTestRunner', task ':example-plugins:rescore:integTestCluster#stop', task ':example-plugins:rescore:integTest', task ':example-plugins:rescore:extractJar', task ':example-plugins:rescore:verifyJarLicense', task ':example-plugins:rescore:verifyJarNotice', task ':example-plugins:rescore:javadoc', task ':example-plugins:rescore:javadocJar', task ':example-plugins:rescore:extractJavadocJar', task ':example-plugins:rescore:verifyJavadocJarLicense', task ':example-plugins:rescore:verifyJavadocJarNotice', task ':example-plugins:rescore:sourcesJar', task ':example-plugins:rescore:extractSourcesJar', task ':example-plugins:rescore:verifySourcesJarLicense', task ':example-plugins:rescore:verifySourcesJarNotice', task ':example-plugins:rescore:check', task ':example-plugins:rescore:build', task ':example-plugins:rest-handler:compileJava', task ':example-plugins:rest-handler:processResources', task ':example-plugins:rest-handler:classes', task ':example-plugins:rest-handler:copyCheckstyleConf', task ':example-plugins:rest-handler:checkstyleMain', task ':example-plugins:rest-handler:copyPluginPropertiesTemplate', task ':example-plugins:rest-handler:pluginProperties', task ':example-plugins:rest-handler:compileTestJava', task ':example-plugins:rest-handler:processTestResources', task ':example-plugins:rest-handler:testClasses', task ':example-plugins:rest-handler:checkstyleTest', task ':example-plugins:rest-handler:forbiddenApisMain', task ':example-plugins:rest-handler:forbiddenApisTest', task ':example-plugins:rest-handler:forbiddenApis', task ':test:fixtures:example-fixture:compileJava', task ':test:fixtures:example-fixture:processResources', task ':test:fixtures:example-fixture:classes', task ':test:fixtures:example-fixture:jar', task ':example-plugins:rest-handler:exampleFixture', task ':example-plugins:rest-handler:jar', task ':example-plugins:rest-handler:bundlePlugin', task ':example-plugins:rest-handler:integTestCluster#prepareCluster.cleanShared', task ':example-plugins:rest-handler:integTestCluster#clean', task ':example-plugins:rest-handler:integTestCluster#createCwd', task ':example-plugins:rest-handler:integTestCluster#checkPrevious', task ':example-plugins:rest-handler:integTestCluster#stopPrevious', task ':example-plugins:rest-handler:integTestCluster#extract', task ':example-plugins:rest-handler:integTestCluster#configure', task ':example-plugins:rest-handler:integTestCluster#copyPlugins', task ':example-plugins:rest-handler:integTestCluster#installRestHandlerPlugin', task ':example-plugins:rest-handler:integTestCluster#start', task ':example-plugins:rest-handler:copyRestSpec', task ':example-plugins:rest-handler:checkstyle', task ':example-plugins:rest-handler:dependencyLicenses', task ':example-plugins:rest-handler:filepermissions', task ':example-plugins:rest-handler:forbiddenPatterns', task ':example-plugins:rest-handler:jarHell', task ':example-plugins:rest-handler:licenseHeaders', task ':example-plugins:rest-handler:loggerUsageCheck', task ':example-plugins:rest-handler:namingConventions', task ':example-plugins:rest-handler:thirdPartyAudit', task ':example-plugins:rest-handler:precommit', task ':test:fixtures:example-fixture:copyCheckstyleConf', task ':test:fixtures:example-fixture:checkstyleMain', task ':test:fixtures:example-fixture:compileTestJava', task ':test:fixtures:example-fixture:processTestResources', task ':test:fixtures:example-fixture:testClasses', task ':test:fixtures:example-fixture:checkstyleTest', task ':test:fixtures:example-fixture:checkstyle', task ':test:fixtures:example-fixture:dependencyLicenses', task ':test:fixtures:example-fixture:filepermissions', task ':test:fixtures:example-fixture:forbiddenApisMain', task ':test:fixtures:example-fixture:forbiddenApisTest', task ':test:fixtures:example-fixture:forbiddenApis', task ':test:fixtures:example-fixture:forbiddenPatterns', task ':test:fixtures:example-fixture:jarHell', task ':test:fixtures:example-fixture:licenseHeaders', task ':test:fixtures:example-fixture:loggerUsageCheck', task ':test:fixtures:example-fixture:namingConventions', task ':test:fixtures:example-fixture:thirdPartyAudit', task ':test:fixtures:example-fixture:precommit', task ':test:fixtures:example-fixture:test', task ':example-plugins:rest-handler:test', task ':example-plugins:rest-handler:integTestCluster#init', task ':example-plugins:rest-handler:integTestCluster#wait', task ':example-plugins:rest-handler:integTestRunner', task ':example-plugins:rest-handler:integTestCluster#stop', task ':example-plugins:rest-handler:exampleFixture#stop', task ':example-plugins:rest-handler:integTest', task ':example-plugins:rest-handler:extractJar', task ':example-plugins:rest-handler:verifyJarLicense', task ':example-plugins:rest-handler:verifyJarNotice', task ':example-plugins:rest-handler:javadoc', task ':example-plugins:rest-handler:javadocJar', task ':example-plugins:rest-handler:extractJavadocJar', task ':example-plugins:rest-handler:verifyJavadocJarLicense', task ':example-plugins:rest-handler:verifyJavadocJarNotice', task ':example-plugins:rest-handler:sourcesJar', task ':example-plugins:rest-handler:extractSourcesJar', task ':example-plugins:rest-handler:verifySourcesJarLicense', task ':example-plugins:rest-handler:verifySourcesJarNotice', task ':example-plugins:rest-handler:check', task ':example-plugins:rest-handler:build', task ':example-plugins:script-expert-scoring:compileJava', task ':example-plugins:script-expert-scoring:processResources', task ':example-plugins:script-expert-scoring:classes', task ':example-plugins:script-expert-scoring:copyCheckstyleConf', task ':example-plugins:script-expert-scoring:checkstyleMain', task ':example-plugins:script-expert-scoring:copyPluginPropertiesTemplate', task ':example-plugins:script-expert-scoring:pluginProperties', task ':example-plugins:script-expert-scoring:compileTestJava', task ':example-plugins:script-expert-scoring:processTestResources', task ':example-plugins:script-expert-scoring:testClasses', task ':example-plugins:script-expert-scoring:checkstyleTest', task ':example-plugins:script-expert-scoring:forbiddenApisMain', task ':example-plugins:script-expert-scoring:forbiddenApisTest', task ':example-plugins:script-expert-scoring:forbiddenApis', task ':example-plugins:script-expert-scoring:jar', task ':example-plugins:script-expert-scoring:bundlePlugin', task ':example-plugins:script-expert-scoring:integTestCluster#prepareCluster.cleanShared', task ':example-plugins:script-expert-scoring:integTestCluster#clean', task ':example-plugins:script-expert-scoring:integTestCluster#createCwd', task ':example-plugins:script-expert-scoring:integTestCluster#checkPrevious', task ':example-plugins:script-expert-scoring:integTestCluster#stopPrevious', task ':example-plugins:script-expert-scoring:integTestCluster#extract', task ':example-plugins:script-expert-scoring:integTestCluster#configure', task ':example-plugins:script-expert-scoring:integTestCluster#copyPlugins', task ':example-plugins:script-expert-scoring:integTestCluster#installScriptExpertScoringPlugin', task ':example-plugins:script-expert-scoring:integTestCluster#start', task ':example-plugins:script-expert-scoring:copyRestSpec', task ':example-plugins:script-expert-scoring:checkstyle', task ':example-plugins:script-expert-scoring:dependencyLicenses', task ':example-plugins:script-expert-scoring:filepermissions', task ':example-plugins:script-expert-scoring:forbiddenPatterns', task ':example-plugins:script-expert-scoring:jarHell', task ':example-plugins:script-expert-scoring:licenseHeaders', task ':example-plugins:script-expert-scoring:loggerUsageCheck', task ':example-plugins:script-expert-scoring:namingConventions', task ':example-plugins:script-expert-scoring:thirdPartyAudit', task ':example-plugins:script-expert-scoring:precommit', task ':example-plugins:script-expert-scoring:test', task ':example-plugins:script-expert-scoring:integTestCluster#init', task ':example-plugins:script-expert-scoring:integTestCluster#wait', task ':example-plugins:script-expert-scoring:integTestRunner', task ':example-plugins:script-expert-scoring:integTestCluster#stop', task ':example-plugins:script-expert-scoring:integTest', task ':example-plugins:script-expert-scoring:extractJar', task ':example-plugins:script-expert-scoring:verifyJarLicense', task ':example-plugins:script-expert-scoring:verifyJarNotice', task ':example-plugins:script-expert-scoring:javadoc', task ':example-plugins:script-expert-scoring:javadocJar', task ':example-plugins:script-expert-scoring:extractJavadocJar', task ':example-plugins:script-expert-scoring:verifyJavadocJarLicense', task ':example-plugins:script-expert-scoring:verifyJavadocJarNotice', task ':example-plugins:script-expert-scoring:sourcesJar', task ':example-plugins:script-expert-scoring:extractSourcesJar', task ':example-plugins:script-expert-scoring:verifySourcesJarLicense', task ':example-plugins:script-expert-scoring:verifySourcesJarNotice', task ':example-plugins:script-expert-scoring:check', task ':example-plugins:script-expert-scoring:build', task ':libs:elasticsearch-core:generatePomFileForNebulaPublication', task ':libs:elasticsearch-core:javadocJar', task ':libs:elasticsearch-core:sourcesJar', task ':libs:elasticsearch-core:assemble', task ':libs:elasticsearch-core:extractJar', task ':libs:elasticsearch-core:verifyJarLicense', task ':libs:elasticsearch-core:verifyJarNotice', task ':libs:elasticsearch-core:extractJavadocJar', task ':libs:elasticsearch-core:verifyJavadocJarLicense', task ':libs:elasticsearch-core:verifyJavadocJarNotice', task ':libs:elasticsearch-core:extractSourcesJar', task ':libs:elasticsearch-core:verifySourcesJarLicense', task ':libs:elasticsearch-core:verifySourcesJarNotice', task ':libs:elasticsearch-core:check', task ':libs:elasticsearch-core:build', task ':libs:elasticsearch-nio:generatePomFileForNebulaPublication', task ':libs:elasticsearch-nio:javadoc', task ':libs:elasticsearch-nio:javadocJar', task ':libs:elasticsearch-nio:sourcesJar', task ':libs:elasticsearch-nio:assemble', task ':libs:elasticsearch-nio:extractJar', task ':libs:elasticsearch-nio:verifyJarLicense', task ':libs:elasticsearch-nio:verifyJarNotice', task ':libs:elasticsearch-nio:extractJavadocJar', task ':libs:elasticsearch-nio:verifyJavadocJarLicense', task ':libs:elasticsearch-nio:verifyJavadocJarNotice', task ':libs:elasticsearch-nio:extractSourcesJar', task ':libs:elasticsearch-nio:verifySourcesJarLicense', task ':libs:elasticsearch-nio:verifySourcesJarNotice', task ':libs:elasticsearch-nio:check', task ':libs:elasticsearch-nio:build', task ':libs:grok:javadoc', task ':libs:grok:javadocJar', task ':libs:grok:sourcesJar', task ':libs:grok:assemble', task ':libs:grok:copyCheckstyleConf', task ':libs:grok:checkstyleMain', task ':libs:grok:compileTestJava', task ':libs:grok:processTestResources', task ':libs:grok:testClasses', task ':libs:grok:checkstyleTest', task ':libs:grok:forbiddenApisMain', task ':libs:grok:forbiddenApisTest', task ':libs:grok:forbiddenApis', task ':libs:grok:checkstyle', task ':libs:grok:dependencyLicenses', task ':libs:grok:filepermissions', task ':libs:grok:forbiddenPatterns', task ':libs:grok:jarHell', task ':libs:grok:licenseHeaders', task ':libs:grok:loggerUsageCheck', task ':libs:grok:namingConventions', task ':libs:grok:thirdPartyAudit', task ':libs:grok:precommit', task ':libs:grok:test', task ':libs:grok:extractJar', task ':libs:grok:verifyJarLicense', task ':libs:grok:verifyJarNotice', task ':libs:grok:extractJavadocJar', task ':libs:grok:verifyJavadocJarLicense', task ':libs:grok:verifyJavadocJarNotice', task ':libs:grok:extractSourcesJar', task ':libs:grok:verifySourcesJarLicense', task ':libs:grok:verifySourcesJarNotice', task ':libs:grok:check', task ':libs:grok:build', task ':libs:plugin-classloader:javadocJar', task ':libs:plugin-classloader:sourcesJar', task ':libs:plugin-classloader:assemble', task ':libs:plugin-classloader:extractJar', task ':libs:plugin-classloader:verifyJarLicense', task ':libs:plugin-classloader:verifyJarNotice', task ':libs:plugin-classloader:extractJavadocJar', task ':libs:plugin-classloader:verifyJavadocJarLicense', task ':libs:plugin-classloader:verifyJavadocJarNotice', task ':libs:plugin-classloader:extractSourcesJar', task ':libs:plugin-classloader:verifySourcesJarLicense', task ':libs:plugin-classloader:verifySourcesJarNotice', task ':libs:plugin-classloader:check', task ':libs:plugin-classloader:build', task ':libs:secure-sm:generatePomFileForNebulaPublication', task ':libs:secure-sm:javadocJar', task ':libs:secure-sm:sourcesJar', task ':libs:secure-sm:assemble', task ':libs:secure-sm:extractJar', task ':libs:secure-sm:verifyJarLicense', task ':libs:secure-sm:verifyJarNotice', task ':libs:secure-sm:extractJavadocJar', task ':libs:secure-sm:verifyJavadocJarLicense', task ':libs:secure-sm:verifyJavadocJarNotice', task ':libs:secure-sm:extractSourcesJar', task ':libs:secure-sm:verifySourcesJarLicense', task ':libs:secure-sm:verifySourcesJarNotice', task ':libs:secure-sm:check', task ':libs:secure-sm:build', task ':libs:x-content:generatePomFileForNebulaPublication', task ':libs:x-content:javadocJar', task ':libs:x-content:sourcesJar', task ':libs:x-content:assemble', task ':libs:x-content:extractJar', task ':libs:x-content:verifyJarLicense', task ':libs:x-content:verifyJarNotice', task ':libs:x-content:extractJavadocJar', task ':libs:x-content:verifyJavadocJarLicense', task ':libs:x-content:verifyJavadocJarNotice', task ':libs:x-content:extractSourcesJar', task ':libs:x-content:verifySourcesJarLicense', task ':libs:x-content:verifySourcesJarNotice', task ':libs:x-content:check', task ':libs:x-content:build', task ':modules:aggs-matrix-stats:generatePomFileForClientJarPublication', task ':modules:aggs-matrix-stats:javadocJar', task ':modules:aggs-matrix-stats:sourcesJar', task ':modules:aggs-matrix-stats:clientJar', task ':modules:aggs-matrix-stats:assemble', task ':modules:aggs-matrix-stats:extractJar', task ':modules:aggs-matrix-stats:verifyJarLicense', task ':modules:aggs-matrix-stats:verifyJarNotice', task ':modules:aggs-matrix-stats:extractJavadocJar', task ':modules:aggs-matrix-stats:verifyJavadocJarLicense', task ':modules:aggs-matrix-stats:verifyJavadocJarNotice', task ':modules:aggs-matrix-stats:extractSourcesJar', task ':modules:aggs-matrix-stats:verifySourcesJarLicense', task ':modules:aggs-matrix-stats:verifySourcesJarNotice', task ':modules:aggs-matrix-stats:check', task ':modules:aggs-matrix-stats:build', task ':modules:analysis-common:javadoc', task ':modules:analysis-common:javadocJar', task ':modules:analysis-common:sourcesJar', task ':modules:analysis-common:assemble', task ':modules:analysis-common:copyCheckstyleConf', task ':modules:analysis-common:checkstyleMain', task ':modules:analysis-common:compileTestJava', task ':modules:analysis-common:processTestResources', task ':modules:analysis-common:testClasses', task ':modules:analysis-common:checkstyleTest', task ':modules:analysis-common:forbiddenApisMain', task ':modules:analysis-common:forbiddenApisTest', task ':modules:analysis-common:forbiddenApis', task ':modules:analysis-common:integTestCluster#prepareCluster.cleanShared', task ':modules:analysis-common:integTestCluster#clean', task ':modules:analysis-common:integTestCluster#createCwd', task ':modules:analysis-common:integTestCluster#checkPrevious', task ':modules:analysis-common:integTestCluster#stopPrevious', task ':modules:analysis-common:integTestCluster#extract', task ':modules:analysis-common:integTestCluster#configure', task ':modules:analysis-common:integTestCluster#installAnalysisCommonModule', task ':modules:analysis-common:integTestCluster#start', task ':modules:analysis-common:copyRestSpec', task ':modules:analysis-common:checkstyle', task ':modules:analysis-common:dependencyLicenses', task ':modules:analysis-common:filepermissions', task ':modules:analysis-common:forbiddenPatterns', task ':modules:analysis-common:jarHell', task ':modules:analysis-common:licenseHeaders', task ':modules:analysis-common:loggerUsageCheck', task ':modules:analysis-common:namingConventions', task ':modules:analysis-common:thirdPartyAudit', task ':modules:analysis-common:precommit', task ':modules:analysis-common:test', task ':modules:analysis-common:integTestCluster#init', task ':modules:analysis-common:integTestCluster#wait', task ':modules:analysis-common:integTestRunner', task ':modules:analysis-common:integTestCluster#stop', task ':modules:analysis-common:integTest', task ':modules:analysis-common:extractJar', task ':modules:analysis-common:verifyJarLicense', task ':modules:analysis-common:verifyJarNotice', task ':modules:analysis-common:extractJavadocJar', task ':modules:analysis-common:verifyJavadocJarLicense', task ':modules:analysis-common:verifyJavadocJarNotice', task ':modules:analysis-common:extractSourcesJar', task ':modules:analysis-common:verifySourcesJarLicense', task ':modules:analysis-common:verifySourcesJarNotice', task ':modules:analysis-common:check', task ':modules:analysis-common:build', task ':modules:ingest-common:javadoc', task ':modules:ingest-common:javadocJar', task ':modules:ingest-common:sourcesJar', task ':modules:ingest-common:assemble', task ':modules:ingest-common:copyCheckstyleConf', task ':modules:ingest-common:checkstyleMain', task ':modules:ingest-common:compileTestJava', task ':modules:ingest-common:processTestResources', task ':modules:ingest-common:testClasses', task ':modules:ingest-common:checkstyleTest', task ':modules:ingest-common:forbiddenApisMain', task ':modules:ingest-common:forbiddenApisTest', task ':modules:ingest-common:forbiddenApis', task ':modules:ingest-common:integTestCluster#prepareCluster.cleanShared', task ':modules:ingest-common:integTestCluster#clean', task ':modules:ingest-common:integTestCluster#createCwd', task ':modules:ingest-common:integTestCluster#checkPrevious', task ':modules:ingest-common:integTestCluster#stopPrevious', task ':modules:ingest-common:integTestCluster#extract', task ':modules:ingest-common:integTestCluster#configure', task ':modules:ingest-common:integTestCluster#installIngestCommonModule', task ':modules:ingest-common:integTestCluster#start', task ':modules:ingest-common:copyRestSpec', task ':modules:ingest-common:checkstyle', task ':modules:ingest-common:dependencyLicenses', task ':modules:ingest-common:filepermissions', task ':modules:ingest-common:forbiddenPatterns', task ':modules:ingest-common:jarHell', task ':modules:ingest-common:licenseHeaders', task ':modules:ingest-common:loggerUsageCheck', task ':modules:ingest-common:namingConventions', task ':modules:ingest-common:thirdPartyAudit', task ':modules:ingest-common:precommit', task ':modules:ingest-common:test', task ':modules:ingest-common:integTestCluster#init', task ':modules:ingest-common:integTestCluster#wait', task ':modules:ingest-common:integTestRunner', task ':modules:ingest-common:integTestCluster#stop', task ':modules:ingest-common:integTest', task ':modules:ingest-common:extractJar', task ':modules:ingest-common:verifyJarLicense', task ':modules:ingest-common:verifyJarNotice', task ':modules:ingest-common:extractJavadocJar', task ':modules:ingest-common:verifyJavadocJarLicense', task ':modules:ingest-common:verifyJavadocJarNotice', task ':modules:ingest-common:extractSourcesJar', task ':modules:ingest-common:verifySourcesJarLicense', task ':modules:ingest-common:verifySourcesJarNotice', task ':modules:ingest-common:check', task ':modules:ingest-common:build', task ':modules:lang-expression:javadoc', task ':modules:lang-expression:javadocJar', task ':modules:lang-expression:sourcesJar', task ':modules:lang-expression:assemble', task ':modules:lang-expression:copyCheckstyleConf', task ':modules:lang-expression:checkstyleMain', task ':modules:lang-expression:compileTestJava', task ':modules:lang-expression:processTestResources', task ':modules:lang-expression:testClasses', task ':modules:lang-expression:checkstyleTest', task ':modules:lang-expression:forbiddenApisMain', task ':modules:lang-expression:forbiddenApisTest', task ':modules:lang-expression:forbiddenApis', task ':modules:lang-expression:integTestCluster#prepareCluster.cleanShared', task ':modules:lang-expression:integTestCluster#clean', task ':modules:lang-expression:integTestCluster#createCwd', task ':modules:lang-expression:integTestCluster#checkPrevious', task ':modules:lang-expression:integTestCluster#stopPrevious', task ':modules:lang-expression:integTestCluster#extract', task ':modules:lang-expression:integTestCluster#configure', task ':modules:lang-expression:integTestCluster#installLangExpressionModule', task ':modules:lang-expression:integTestCluster#start', task ':modules:lang-expression:copyRestSpec', task ':modules:lang-expression:checkstyle', task ':modules:lang-expression:dependencyLicenses', task ':modules:lang-expression:filepermissions', task ':modules:lang-expression:forbiddenPatterns', task ':modules:lang-expression:jarHell', task ':modules:lang-expression:licenseHeaders', task ':modules:lang-expression:loggerUsageCheck', task ':modules:lang-expression:namingConventions', task ':modules:lang-expression:thirdPartyAudit', task ':modules:lang-expression:precommit', task ':modules:lang-expression:test', task ':modules:lang-expression:integTestCluster#init', task ':modules:lang-expression:integTestCluster#wait', task ':modules:lang-expression:integTestRunner', task ':modules:lang-expression:integTestCluster#stop', task ':modules:lang-expression:integTest', task ':modules:lang-expression:extractJar', task ':modules:lang-expression:verifyJarLicense', task ':modules:lang-expression:verifyJarNotice', task ':modules:lang-expression:extractJavadocJar', task ':modules:lang-expression:verifyJavadocJarLicense', task ':modules:lang-expression:verifyJavadocJarNotice', task ':modules:lang-expression:extractSourcesJar', task ':modules:lang-expression:verifySourcesJarLicense', task ':modules:lang-expression:verifySourcesJarNotice', task ':modules:lang-expression:check', task ':modules:lang-expression:build', task ':modules:lang-mustache:generatePomFileForClientJarPublication', task ':modules:lang-mustache:javadocJar', task ':modules:lang-mustache:sourcesJar', task ':modules:lang-mustache:clientJar', task ':modules:lang-mustache:assemble', task ':modules:lang-mustache:integTestCluster#prepareCluster.cleanShared', task ':modules:lang-mustache:integTestCluster#clean', task ':modules:lang-mustache:integTestCluster#createCwd', task ':modules:lang-mustache:integTestCluster#checkPrevious', task ':modules:lang-mustache:integTestCluster#stopPrevious', task ':modules:lang-mustache:integTestCluster#extract', task ':modules:lang-mustache:integTestCluster#configure', task ':modules:lang-mustache:integTestCluster#installLangMustacheModule', task ':modules:lang-mustache:integTestCluster#start', task ':modules:lang-mustache:copyRestSpec', task ':modules:lang-mustache:integTestCluster#init', task ':modules:lang-mustache:integTestCluster#wait', task ':modules:lang-mustache:integTestRunner', task ':modules:lang-mustache:integTestCluster#stop', task ':modules:lang-mustache:integTest', task ':modules:lang-mustache:extractJar', task ':modules:lang-mustache:verifyJarLicense', task ':modules:lang-mustache:verifyJarNotice', task ':modules:lang-mustache:extractJavadocJar', task ':modules:lang-mustache:verifyJavadocJarLicense', task ':modules:lang-mustache:verifyJavadocJarNotice', task ':modules:lang-mustache:extractSourcesJar', task ':modules:lang-mustache:verifySourcesJarLicense', task ':modules:lang-mustache:verifySourcesJarNotice', task ':modules:lang-mustache:check', task ':modules:lang-mustache:build', task ':modules:lang-painless:apiJavadoc', task ':modules:lang-painless:apiJavadocJar', task ':modules:lang-painless:spi:javadoc', task ':modules:lang-painless:javadoc', task ':modules:lang-painless:javadocJar', task ':modules:lang-painless:sourcesJar', task ':modules:lang-painless:assemble', task ':modules:lang-painless:extractApiJavadocJar', task ':modules:lang-painless:verifyApiJavadocJarLicense', task ':modules:lang-painless:verifyApiJavadocJarNotice', task ':modules:lang-painless:extractJar', task ':modules:lang-painless:verifyJarLicense', task ':modules:lang-painless:verifyJarNotice', task ':modules:lang-painless:extractJavadocJar', task ':modules:lang-painless:verifyJavadocJarLicense', task ':modules:lang-painless:verifyJavadocJarNotice', task ':modules:lang-painless:extractSourcesJar', task ':modules:lang-painless:verifySourcesJarLicense', task ':modules:lang-painless:verifySourcesJarNotice', task ':modules:lang-painless:check', task ':modules:lang-painless:build', task ':modules:mapper-extras:javadoc', task ':modules:mapper-extras:javadocJar', task ':modules:mapper-extras:sourcesJar', task ':modules:mapper-extras:assemble', task ':modules:mapper-extras:copyCheckstyleConf', task ':modules:mapper-extras:checkstyleMain', task ':modules:mapper-extras:compileTestJava', task ':modules:mapper-extras:processTestResources', task ':modules:mapper-extras:testClasses', task ':modules:mapper-extras:checkstyleTest', task ':modules:mapper-extras:forbiddenApisMain', task ':modules:mapper-extras:forbiddenApisTest', task ':modules:mapper-extras:forbiddenApis', task ':modules:mapper-extras:integTestCluster#prepareCluster.cleanShared', task ':modules:mapper-extras:integTestCluster#clean', task ':modules:mapper-extras:integTestCluster#createCwd', task ':modules:mapper-extras:integTestCluster#checkPrevious', task ':modules:mapper-extras:integTestCluster#stopPrevious', task ':modules:mapper-extras:integTestCluster#extract', task ':modules:mapper-extras:integTestCluster#configure', task ':modules:mapper-extras:integTestCluster#installMapperExtrasModule', task ':modules:mapper-extras:integTestCluster#start', task ':modules:mapper-extras:copyRestSpec', task ':modules:mapper-extras:checkstyle', task ':modules:mapper-extras:dependencyLicenses', task ':modules:mapper-extras:filepermissions', task ':modules:mapper-extras:forbiddenPatterns', task ':modules:mapper-extras:jarHell', task ':modules:mapper-extras:licenseHeaders', task ':modules:mapper-extras:loggerUsageCheck', task ':modules:mapper-extras:namingConventions', task ':modules:mapper-extras:thirdPartyAudit', task ':modules:mapper-extras:precommit', task ':modules:mapper-extras:test', task ':modules:mapper-extras:integTestCluster#init', task ':modules:mapper-extras:integTestCluster#wait', task ':modules:mapper-extras:integTestRunner', task ':modules:mapper-extras:integTestCluster#stop', task ':modules:mapper-extras:integTest', task ':modules:mapper-extras:extractJar', task ':modules:mapper-extras:verifyJarLicense', task ':modules:mapper-extras:verifyJarNotice', task ':modules:mapper-extras:extractJavadocJar', task ':modules:mapper-extras:verifyJavadocJarLicense', task ':modules:mapper-extras:verifyJavadocJarNotice', task ':modules:mapper-extras:extractSourcesJar', task ':modules:mapper-extras:verifySourcesJarLicense', task ':modules:mapper-extras:verifySourcesJarNotice', task ':modules:mapper-extras:check', task ':modules:mapper-extras:build', task ':modules:parent-join:generatePomFileForClientJarPublication', task ':modules:parent-join:javadocJar', task ':modules:parent-join:sourcesJar', task ':modules:parent-join:clientJar', task ':modules:parent-join:assemble', task ':modules:parent-join:extractJar', task ':modules:parent-join:verifyJarLicense', task ':modules:parent-join:verifyJarNotice', task ':modules:parent-join:extractJavadocJar', task ':modules:parent-join:verifyJavadocJarLicense', task ':modules:parent-join:verifyJavadocJarNotice', task ':modules:parent-join:extractSourcesJar', task ':modules:parent-join:verifySourcesJarLicense', task ':modules:parent-join:verifySourcesJarNotice', task ':modules:parent-join:check', task ':modules:parent-join:build', task ':modules:percolator:generatePomFileForClientJarPublication', task ':modules:percolator:javadocJar', task ':modules:percolator:sourcesJar', task ':modules:percolator:clientJar', task ':modules:percolator:assemble', task ':modules:percolator:integTestCluster#prepareCluster.cleanShared', task ':modules:percolator:integTestCluster#clean', task ':modules:percolator:integTestCluster#createCwd', task ':modules:percolator:integTestCluster#checkPrevious', task ':modules:percolator:integTestCluster#stopPrevious', task ':modules:percolator:integTestCluster#extract', task ':modules:percolator:integTestCluster#configure', task ':modules:percolator:integTestCluster#installPercolatorModule', task ':modules:percolator:integTestCluster#start', task ':modules:percolator:copyRestSpec', task ':modules:percolator:integTestCluster#init', task ':modules:percolator:integTestCluster#wait', task ':modules:percolator:integTestRunner', task ':modules:percolator:integTestCluster#stop', task ':modules:percolator:integTest', task ':modules:percolator:extractJar', task ':modules:percolator:verifyJarLicense', task ':modules:percolator:verifyJarNotice', task ':modules:percolator:extractJavadocJar', task ':modules:percolator:verifyJavadocJarLicense', task ':modules:percolator:verifyJavadocJarNotice', task ':modules:percolator:extractSourcesJar', task ':modules:percolator:verifySourcesJarLicense', task ':modules:percolator:verifySourcesJarNotice', task ':modules:percolator:check', task ':modules:percolator:build', task ':modules:rank-eval:generatePomFileForClientJarPublication', task ':modules:rank-eval:javadocJar', task ':modules:rank-eval:sourcesJar', task ':modules:rank-eval:clientJar', task ':modules:rank-eval:assemble', task ':modules:rank-eval:extractJar', task ':modules:rank-eval:verifyJarLicense', task ':modules:rank-eval:verifyJarNotice', task ':modules:rank-eval:extractJavadocJar', task ':modules:rank-eval:verifyJavadocJarLicense', task ':modules:rank-eval:verifyJavadocJarNotice', task ':modules:rank-eval:extractSourcesJar', task ':modules:rank-eval:verifySourcesJarLicense', task ':modules:rank-eval:verifySourcesJarNotice', task ':modules:rank-eval:check', task ':modules:rank-eval:build', task ':modules:reindex:generatePomFileForClientJarPublication', task ':modules:reindex:javadocJar', task ':modules:reindex:sourcesJar', task ':modules:reindex:clientJar', task ':modules:reindex:assemble', task ':modules:reindex:integTestCluster#prepareCluster.cleanShared', task ':modules:reindex:integTestCluster#clean', task ':modules:reindex:integTestCluster#createCwd', task ':modules:reindex:integTestCluster#checkPrevious', task ':modules:reindex:integTestCluster#stopPrevious', task ':modules:reindex:integTestCluster#extract', task ':modules:reindex:integTestCluster#configure', task ':modules:reindex:integTestCluster#installReindexModule', task ':modules:reindex:integTestCluster#start', task ':modules:reindex:copyRestSpec', task ':modules:transport-netty4:integTestCluster#prepareCluster.cleanShared', task ':modules:transport-netty4:integTestCluster#clean', task ':modules:transport-netty4:integTestCluster#createCwd', task ':modules:transport-netty4:integTestCluster#checkPrevious', task ':modules:transport-netty4:integTestCluster#stopPrevious', task ':modules:transport-netty4:integTestCluster#extract', task ':modules:transport-netty4:integTestCluster#configure', task ':modules:transport-netty4:integTestCluster#installTransportNetty4Module', task ':modules:transport-netty4:integTestCluster#start', task ':modules:transport-netty4:copyRestSpec', task ':modules:transport-netty4:integTestCluster#init', task ':modules:transport-netty4:integTestCluster#wait', task ':modules:transport-netty4:integTestRunner', task ':modules:transport-netty4:integTestCluster#stop', task ':modules:transport-netty4:integTest', task ':modules:reindex:integTestCluster#init', task ':modules:reindex:integTestCluster#wait', task ':modules:reindex:integTestRunner', task ':modules:reindex:integTestCluster#stop', task ':modules:reindex:integTest', task ':modules:reindex:extractJar', task ':modules:reindex:verifyJarLicense', task ':modules:reindex:verifyJarNotice', task ':modules:reindex:extractJavadocJar', task ':modules:reindex:verifyJavadocJarLicense', task ':modules:reindex:verifyJavadocJarNotice', task ':modules:reindex:extractSourcesJar', task ':modules:reindex:verifySourcesJarLicense', task ':modules:reindex:verifySourcesJarNotice', task ':modules:reindex:check', task ':modules:reindex:build', task ':modules:repository-url:javadoc', task ':modules:repository-url:javadocJar', task ':modules:repository-url:sourcesJar', task ':modules:repository-url:assemble', task ':modules:repository-url:copyCheckstyleConf', task ':modules:repository-url:checkstyleMain', task ':modules:repository-url:compileTestJava', task ':modules:repository-url:processTestResources', task ':modules:repository-url:testClasses', task ':modules:repository-url:checkstyleTest', task ':modules:repository-url:forbiddenApisMain', task ':modules:repository-url:forbiddenApisTest', task ':modules:repository-url:forbiddenApis', task ':modules:repository-url:urlFixture', task ':modules:repository-url:integTestCluster#prepareCluster.cleanShared', task ':modules:repository-url:integTestCluster#clean', task ':modules:repository-url:integTestCluster#createCwd', task ':modules:repository-url:integTestCluster#checkPrevious', task ':modules:repository-url:integTestCluster#stopPrevious', task ':modules:repository-url:integTestCluster#extract', task ':modules:repository-url:integTestCluster#configure', task ':modules:repository-url:integTestCluster#installRepositoryUrlModule', task ':modules:repository-url:integTestCluster#start', task ':modules:repository-url:copyRestSpec', task ':modules:repository-url:checkstyle', task ':modules:repository-url:dependencyLicenses', task ':modules:repository-url:filepermissions', task ':modules:repository-url:forbiddenPatterns', task ':modules:repository-url:jarHell', task ':modules:repository-url:licenseHeaders', task ':modules:repository-url:loggerUsageCheck', task ':modules:repository-url:namingConventions', task ':modules:repository-url:thirdPartyAudit', task ':modules:repository-url:precommit', task ':modules:repository-url:test', task ':modules:repository-url:integTestCluster#init', task ':modules:repository-url:integTestCluster#wait', task ':modules:repository-url:integTestRunner', task ':modules:repository-url:urlFixture#stop', task ':modules:repository-url:integTestCluster#stop', task ':modules:repository-url:integTest', task ':modules:repository-url:extractJar', task ':modules:repository-url:verifyJarLicense', task ':modules:repository-url:verifyJarNotice', task ':modules:repository-url:extractJavadocJar', task ':modules:repository-url:verifyJavadocJarLicense', task ':modules:repository-url:verifyJavadocJarNotice', task ':modules:repository-url:extractSourcesJar', task ':modules:repository-url:verifySourcesJarLicense', task ':modules:repository-url:verifySourcesJarNotice', task ':modules:repository-url:check', task ':modules:repository-url:build', task ':modules:transport-netty4:generatePomFileForClientJarPublication', task ':modules:transport-netty4:javadocJar', task ':modules:transport-netty4:sourcesJar', task ':modules:transport-netty4:clientJar', task ':modules:transport-netty4:assemble', task ':modules:transport-netty4:extractJar', task ':modules:transport-netty4:verifyJarLicense', task ':modules:transport-netty4:verifyJarNotice', task ':modules:transport-netty4:extractJavadocJar', task ':modules:transport-netty4:verifyJavadocJarLicense', task ':modules:transport-netty4:verifyJavadocJarNotice', task ':modules:transport-netty4:extractSourcesJar', task ':modules:transport-netty4:verifySourcesJarLicense', task ':modules:transport-netty4:verifySourcesJarNotice', task ':modules:transport-netty4:check', task ':modules:transport-netty4:build', task ':plugins:analysis-icu:generatePomFileForZipPublication', task ':plugins:analysis-icu:generatePomFileForZipRealPublication', task ':plugins:analysis-icu:javadoc', task ':plugins:analysis-icu:javadocJar', task ':plugins:analysis-icu:sourcesJar', task ':plugins:analysis-icu:assemble', task ':plugins:analysis-icu:extractJar', task ':plugins:analysis-icu:verifyJarLicense', task ':plugins:analysis-icu:verifyJarNotice', task ':plugins:analysis-icu:extractJavadocJar', task ':plugins:analysis-icu:verifyJavadocJarLicense', task ':plugins:analysis-icu:verifyJavadocJarNotice', task ':plugins:analysis-icu:extractSourcesJar', task ':plugins:analysis-icu:verifySourcesJarLicense', task ':plugins:analysis-icu:verifySourcesJarNotice', task ':plugins:analysis-icu:check', task ':plugins:analysis-icu:build', task ':plugins:analysis-kuromoji:generatePomFileForZipPublication', task ':plugins:analysis-kuromoji:generatePomFileForZipRealPublication', task ':plugins:analysis-kuromoji:javadoc', task ':plugins:analysis-kuromoji:javadocJar', task ':plugins:analysis-kuromoji:sourcesJar', task ':plugins:analysis-kuromoji:assemble', task ':plugins:analysis-kuromoji:extractJar', task ':plugins:analysis-kuromoji:verifyJarLicense', task ':plugins:analysis-kuromoji:verifyJarNotice', task ':plugins:analysis-kuromoji:extractJavadocJar', task ':plugins:analysis-kuromoji:verifyJavadocJarLicense', task ':plugins:analysis-kuromoji:verifyJavadocJarNotice', task ':plugins:analysis-kuromoji:extractSourcesJar', task ':plugins:analysis-kuromoji:verifySourcesJarLicense', task ':plugins:analysis-kuromoji:verifySourcesJarNotice', task ':plugins:analysis-kuromoji:check', task ':plugins:analysis-kuromoji:build', task ':plugins:analysis-nori:generatePomFileForZipPublication', task ':plugins:analysis-nori:generatePomFileForZipRealPublication', task ':plugins:analysis-nori:javadoc', task ':plugins:analysis-nori:javadocJar', task ':plugins:analysis-nori:sourcesJar', task ':plugins:analysis-nori:assemble', task ':plugins:analysis-nori:extractJar', task ':plugins:analysis-nori:verifyJarLicense', task ':plugins:analysis-nori:verifyJarNotice', task ':plugins:analysis-nori:extractJavadocJar', task ':plugins:analysis-nori:verifyJavadocJarLicense', task ':plugins:analysis-nori:verifyJavadocJarNotice', task ':plugins:analysis-nori:extractSourcesJar', task ':plugins:analysis-nori:verifySourcesJarLicense', task ':plugins:analysis-nori:verifySourcesJarNotice', task ':plugins:analysis-nori:check', task ':plugins:analysis-nori:build', task ':plugins:analysis-phonetic:generatePomFileForZipPublication', task ':plugins:analysis-phonetic:generatePomFileForZipRealPublication', task ':plugins:analysis-phonetic:javadoc', task ':plugins:analysis-phonetic:javadocJar', task ':plugins:analysis-phonetic:sourcesJar', task ':plugins:analysis-phonetic:assemble', task ':plugins:analysis-phonetic:extractJar', task ':plugins:analysis-phonetic:verifyJarLicense', task ':plugins:analysis-phonetic:verifyJarNotice', task ':plugins:analysis-phonetic:extractJavadocJar', task ':plugins:analysis-phonetic:verifyJavadocJarLicense', task ':plugins:analysis-phonetic:verifyJavadocJarNotice', task ':plugins:analysis-phonetic:extractSourcesJar', task ':plugins:analysis-phonetic:verifySourcesJarLicense', task ':plugins:analysis-phonetic:verifySourcesJarNotice', task ':plugins:analysis-phonetic:check', task ':plugins:analysis-phonetic:build', task ':plugins:analysis-smartcn:generatePomFileForZipPublication', task ':plugins:analysis-smartcn:generatePomFileForZipRealPublication', task ':plugins:analysis-smartcn:javadoc', task ':plugins:analysis-smartcn:javadocJar', task ':plugins:analysis-smartcn:sourcesJar', task ':plugins:analysis-smartcn:assemble', task ':plugins:analysis-smartcn:extractJar', task ':plugins:analysis-smartcn:verifyJarLicense', task ':plugins:analysis-smartcn:verifyJarNotice', task ':plugins:analysis-smartcn:extractJavadocJar', task ':plugins:analysis-smartcn:verifyJavadocJarLicense', task ':plugins:analysis-smartcn:verifyJavadocJarNotice', task ':plugins:analysis-smartcn:extractSourcesJar', task ':plugins:analysis-smartcn:verifySourcesJarLicense', task ':plugins:analysis-smartcn:verifySourcesJarNotice', task ':plugins:analysis-smartcn:check', task ':plugins:analysis-smartcn:build', task ':plugins:analysis-stempel:generatePomFileForZipPublication', task ':plugins:analysis-stempel:generatePomFileForZipRealPublication', task ':plugins:analysis-stempel:javadoc', task ':plugins:analysis-stempel:javadocJar', task ':plugins:analysis-stempel:sourcesJar', task ':plugins:analysis-stempel:assemble', task ':plugins:analysis-stempel:extractJar', task ':plugins:analysis-stempel:verifyJarLicense', task ':plugins:analysis-stempel:verifyJarNotice', task ':plugins:analysis-stempel:extractJavadocJar', task ':plugins:analysis-stempel:verifyJavadocJarLicense', task ':plugins:analysis-stempel:verifyJavadocJarNotice', task ':plugins:analysis-stempel:extractSourcesJar', task ':plugins:analysis-stempel:verifySourcesJarLicense', task ':plugins:analysis-stempel:verifySourcesJarNotice', task ':plugins:analysis-stempel:check', task ':plugins:analysis-stempel:build', task ':plugins:analysis-ukrainian:generatePomFileForZipPublication', task ':plugins:analysis-ukrainian:generatePomFileForZipRealPublication', task ':plugins:analysis-ukrainian:javadoc', task ':plugins:analysis-ukrainian:javadocJar', task ':plugins:analysis-ukrainian:sourcesJar', task ':plugins:analysis-ukrainian:assemble', task ':plugins:analysis-ukrainian:extractJar', task ':plugins:analysis-ukrainian:verifyJarLicense', task ':plugins:analysis-ukrainian:verifyJarNotice', task ':plugins:analysis-ukrainian:extractJavadocJar', task ':plugins:analysis-ukrainian:verifyJavadocJarLicense', task ':plugins:analysis-ukrainian:verifyJavadocJarNotice', task ':plugins:analysis-ukrainian:extractSourcesJar', task ':plugins:analysis-ukrainian:verifySourcesJarLicense', task ':plugins:analysis-ukrainian:verifySourcesJarNotice', task ':plugins:analysis-ukrainian:check', task ':plugins:analysis-ukrainian:build', task ':plugins:discovery-azure-classic:generatePomFileForZipPublication', task ':plugins:discovery-azure-classic:generatePomFileForZipRealPublication', task ':plugins:discovery-azure-classic:javadoc', task ':plugins:discovery-azure-classic:javadocJar', task ':plugins:discovery-azure-classic:sourcesJar', task ':plugins:discovery-azure-classic:assemble', task ':plugins:discovery-azure-classic:extractJar', task ':plugins:discovery-azure-classic:verifyJarLicense', task ':plugins:discovery-azure-classic:verifyJarNotice', task ':plugins:discovery-azure-classic:extractJavadocJar', task ':plugins:discovery-azure-classic:verifyJavadocJarLicense', task ':plugins:discovery-azure-classic:verifyJavadocJarNotice', task ':plugins:discovery-azure-classic:extractSourcesJar', task ':plugins:discovery-azure-classic:verifySourcesJarLicense', task ':plugins:discovery-azure-classic:verifySourcesJarNotice', task ':plugins:discovery-azure-classic:check', task ':plugins:discovery-azure-classic:build', task ':plugins:discovery-ec2:generatePomFileForZipPublication', task ':plugins:discovery-ec2:generatePomFileForZipRealPublication', task ':plugins:discovery-ec2:javadoc', task ':plugins:discovery-ec2:javadocJar', task ':plugins:discovery-ec2:sourcesJar', task ':plugins:discovery-ec2:assemble', task ':plugins:discovery-ec2:extractJar', task ':plugins:discovery-ec2:verifyJarLicense', task ':plugins:discovery-ec2:verifyJarNotice', task ':plugins:discovery-ec2:extractJavadocJar', task ':plugins:discovery-ec2:verifyJavadocJarLicense', task ':plugins:discovery-ec2:verifyJavadocJarNotice', task ':plugins:discovery-ec2:extractSourcesJar', task ':plugins:discovery-ec2:verifySourcesJarLicense', task ':plugins:discovery-ec2:verifySourcesJarNotice', task ':plugins:discovery-ec2:check', task ':plugins:discovery-ec2:build', task ':plugins:discovery-file:generatePomFileForZipPublication', task ':plugins:discovery-file:generatePomFileForZipRealPublication', task ':plugins:discovery-file:javadoc', task ':plugins:discovery-file:javadocJar', task ':plugins:discovery-file:sourcesJar', task ':plugins:discovery-file:assemble', task ':plugins:discovery-file:extractJar', task ':plugins:discovery-file:verifyJarLicense', task ':plugins:discovery-file:verifyJarNotice', task ':plugins:discovery-file:extractJavadocJar', task ':plugins:discovery-file:verifyJavadocJarLicense', task ':plugins:discovery-file:verifyJavadocJarNotice', task ':plugins:discovery-file:extractSourcesJar', task ':plugins:discovery-file:verifySourcesJarLicense', task ':plugins:discovery-file:verifySourcesJarNotice', task ':plugins:discovery-file:check', task ':plugins:discovery-file:build', task ':plugins:discovery-gce:generatePomFileForZipPublication', task ':plugins:discovery-gce:generatePomFileForZipRealPublication', task ':plugins:discovery-gce:javadoc', task ':plugins:discovery-gce:javadocJar', task ':plugins:discovery-gce:sourcesJar', task ':plugins:discovery-gce:assemble', task ':plugins:discovery-gce:extractJar', task ':plugins:discovery-gce:verifyJarLicense', task ':plugins:discovery-gce:verifyJarNotice', task ':plugins:discovery-gce:extractJavadocJar', task ':plugins:discovery-gce:verifyJavadocJarLicense', task ':plugins:discovery-gce:verifyJavadocJarNotice', task ':plugins:discovery-gce:extractSourcesJar', task ':plugins:discovery-gce:verifySourcesJarLicense', task ':plugins:discovery-gce:verifySourcesJarNotice', task ':plugins:discovery-gce:check', task ':plugins:discovery-gce:build', task ':plugins:ingest-attachment:generatePomFileForZipPublication', task ':plugins:ingest-attachment:generatePomFileForZipRealPublication', task ':plugins:ingest-attachment:javadoc', task ':plugins:ingest-attachment:javadocJar', task ':plugins:ingest-attachment:sourcesJar', task ':plugins:ingest-attachment:assemble', task ':plugins:ingest-attachment:extractJar', task ':plugins:ingest-attachment:verifyJarLicense', task ':plugins:ingest-attachment:verifyJarNotice', task ':plugins:ingest-attachment:extractJavadocJar', task ':plugins:ingest-attachment:verifyJavadocJarLicense', task ':plugins:ingest-attachment:verifyJavadocJarNotice', task ':plugins:ingest-attachment:extractSourcesJar', task ':plugins:ingest-attachment:verifySourcesJarLicense', task ':plugins:ingest-attachment:verifySourcesJarNotice', task ':plugins:ingest-attachment:check', task ':plugins:ingest-attachment:build', task ':plugins:ingest-geoip:generatePomFileForZipPublication', task ':plugins:ingest-geoip:generatePomFileForZipRealPublication', task ':plugins:ingest-geoip:javadoc', task ':plugins:ingest-geoip:javadocJar', task ':plugins:ingest-geoip:sourcesJar', task ':plugins:ingest-geoip:assemble', task ':plugins:ingest-geoip:extractJar', task ':plugins:ingest-geoip:verifyJarLicense', task ':plugins:ingest-geoip:verifyJarNotice', task ':plugins:ingest-geoip:extractJavadocJar', task ':plugins:ingest-geoip:verifyJavadocJarLicense', task ':plugins:ingest-geoip:verifyJavadocJarNotice', task ':plugins:ingest-geoip:extractSourcesJar', task ':plugins:ingest-geoip:verifySourcesJarLicense', task ':plugins:ingest-geoip:verifySourcesJarNotice', task ':plugins:ingest-geoip:check', task ':plugins:ingest-geoip:build', task ':plugins:ingest-user-agent:generatePomFileForZipPublication', task ':plugins:ingest-user-agent:generatePomFileForZipRealPublication', task ':plugins:ingest-user-agent:javadoc', task ':plugins:ingest-user-agent:javadocJar', task ':plugins:ingest-user-agent:sourcesJar', task ':plugins:ingest-user-agent:assemble', task ':plugins:ingest-user-agent:extractJar', task ':plugins:ingest-user-agent:verifyJarLicense', task ':plugins:ingest-user-agent:verifyJarNotice', task ':plugins:ingest-user-agent:extractJavadocJar', task ':plugins:ingest-user-agent:verifyJavadocJarLicense', task ':plugins:ingest-user-agent:verifyJavadocJarNotice', task ':plugins:ingest-user-agent:extractSourcesJar', task ':plugins:ingest-user-agent:verifySourcesJarLicense', task ':plugins:ingest-user-agent:verifySourcesJarNotice', task ':plugins:ingest-user-agent:check', task ':plugins:ingest-user-agent:build', task ':plugins:mapper-murmur3:generatePomFileForZipPublication', task ':plugins:mapper-murmur3:generatePomFileForZipRealPublication', task ':plugins:mapper-murmur3:javadoc', task ':plugins:mapper-murmur3:javadocJar', task ':plugins:mapper-murmur3:sourcesJar', task ':plugins:mapper-murmur3:assemble', task ':plugins:mapper-murmur3:extractJar', task ':plugins:mapper-murmur3:verifyJarLicense', task ':plugins:mapper-murmur3:verifyJarNotice', task ':plugins:mapper-murmur3:extractJavadocJar', task ':plugins:mapper-murmur3:verifyJavadocJarLicense', task ':plugins:mapper-murmur3:verifyJavadocJarNotice', task ':plugins:mapper-murmur3:extractSourcesJar', task ':plugins:mapper-murmur3:verifySourcesJarLicense', task ':plugins:mapper-murmur3:verifySourcesJarNotice', task ':plugins:mapper-murmur3:check', task ':plugins:mapper-murmur3:build', task ':plugins:mapper-size:generatePomFileForZipPublication', task ':plugins:mapper-size:generatePomFileForZipRealPublication', task ':plugins:mapper-size:javadoc', task ':plugins:mapper-size:javadocJar', task ':plugins:mapper-size:sourcesJar', task ':plugins:mapper-size:assemble', task ':plugins:mapper-size:extractJar', task ':plugins:mapper-size:verifyJarLicense', task ':plugins:mapper-size:verifyJarNotice', task ':plugins:mapper-size:extractJavadocJar', task ':plugins:mapper-size:verifyJavadocJarLicense', task ':plugins:mapper-size:verifyJavadocJarNotice', task ':plugins:mapper-size:extractSourcesJar', task ':plugins:mapper-size:verifySourcesJarLicense', task ':plugins:mapper-size:verifySourcesJarNotice', task ':plugins:mapper-size:check', task ':plugins:mapper-size:build', task ':plugins:repository-azure:generateNotice', task ':plugins:repository-azure:compileJava', task ':plugins:repository-azure:processResources', task ':plugins:repository-azure:classes', task ':plugins:repository-azure:jar', task ':plugins:repository-azure:copyPluginPropertiesTemplate', task ':plugins:repository-azure:pluginProperties', task ':plugins:repository-azure:bundlePlugin', task ':plugins:repository-azure:generatePomFileForZipPublication', task ':plugins:repository-azure:generatePomFileForZipRealPublication', task ':plugins:repository-azure:javadoc', task ':plugins:repository-azure:javadocJar', task ':plugins:repository-azure:sourcesJar', task ':plugins:repository-azure:assemble', task ':plugins:repository-azure:copyCheckstyleConf', task ':plugins:repository-azure:checkstyleMain', task ':plugins:repository-azure:compileTestJava', task ':plugins:repository-azure:processTestResources', task ':plugins:repository-azure:testClasses', task ':plugins:repository-azure:checkstyleTest', task ':plugins:repository-azure:forbiddenApisMain', task ':plugins:repository-azure:forbiddenApisTest', task ':plugins:repository-azure:forbiddenApis', task ':plugins:repository-azure:integTestCluster#prepareCluster.cleanShared', task ':plugins:repository-azure:integTestCluster#clean', task ':plugins:repository-azure:integTestCluster#createCwd', task ':plugins:repository-azure:integTestCluster#checkPrevious', task ':plugins:repository-azure:integTestCluster#stopPrevious', task ':plugins:repository-azure:integTestCluster#extract', task ':plugins:repository-azure:integTestCluster#configure', task ':plugins:repository-azure:integTestCluster#createKeystore', task ':plugins:repository-azure:integTestCluster#addToKeystore#azure.client.integration_test.account', task ':plugins:repository-azure:integTestCluster#addToKeystore#azure.client.integration_test.key', task ':plugins:repository-azure:integTestCluster#copyPlugins', task ':plugins:repository-azure:integTestCluster#installRepositoryAzurePlugin', task ':plugins:repository-azure:integTestCluster#start', task ':plugins:repository-azure:copyRestSpec', task ':plugins:repository-azure:checkstyle', task ':plugins:repository-azure:dependencyLicenses', task ':plugins:repository-azure:filepermissions', task ':plugins:repository-azure:forbiddenPatterns', task ':plugins:repository-azure:jarHell', task ':plugins:repository-azure:licenseHeaders', task ':plugins:repository-azure:loggerUsageCheck', task ':plugins:repository-azure:namingConventions', task ':plugins:repository-azure:thirdPartyAudit', task ':plugins:repository-azure:precommit', task ':plugins:repository-azure:test', task ':plugins:repository-azure:integTestCluster#init', task ':plugins:repository-azure:integTestCluster#wait', task ':plugins:repository-azure:integTestRunner', task ':plugins:repository-azure:integTestCluster#stop', task ':plugins:repository-azure:integTest', task ':plugins:repository-azure:extractJar', task ':plugins:repository-azure:verifyJarLicense', task ':plugins:repository-azure:verifyJarNotice', task ':plugins:repository-azure:extractJavadocJar', task ':plugins:repository-azure:verifyJavadocJarLicense', task ':plugins:repository-azure:verifyJavadocJarNotice', task ':plugins:repository-azure:extractSourcesJar', task ':plugins:repository-azure:verifySourcesJarLicense', task ':plugins:repository-azure:verifySourcesJarNotice', task ':plugins:repository-azure:qa:microsoft-azure-storage:copyCheckstyleConf', task ':plugins:repository-azure:qa:microsoft-azure-storage:compileTestJava', task ':plugins:repository-azure:qa:microsoft-azure-storage:processTestResources', task ':plugins:repository-azure:qa:microsoft-azure-storage:testClasses', task ':plugins:repository-azure:qa:microsoft-azure-storage:checkstyleTest', task ':plugins:repository-azure:qa:microsoft-azure-storage:forbiddenApisTest', task ':plugins:repository-azure:qa:microsoft-azure-storage:forbiddenApis', task ':plugins:repository-azure:qa:microsoft-azure-storage:azureStorageFixture', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestCluster#prepareCluster.cleanShared', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestCluster#clean', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestCluster#createCwd', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestCluster#checkPrevious', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestCluster#stopPrevious', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestCluster#extract', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestCluster#configure', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestCluster#createKeystore', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestCluster#addToKeystore#azure.client.integration_test.account', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestCluster#addToKeystore#azure.client.integration_test.key', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestCluster#copyPlugins', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestCluster#installRepositoryAzurePlugin', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestCluster#start', task ':plugins:repository-azure:qa:microsoft-azure-storage:copyRestSpec', task ':plugins:repository-azure:qa:microsoft-azure-storage:checkstyle', task ':plugins:repository-azure:qa:microsoft-azure-storage:filepermissions', task ':plugins:repository-azure:qa:microsoft-azure-storage:forbiddenPatterns', task ':plugins:repository-azure:qa:microsoft-azure-storage:jarHell', task ':plugins:repository-azure:qa:microsoft-azure-storage:licenseHeaders', task ':plugins:repository-azure:qa:microsoft-azure-storage:loggerUsageCheck', task ':plugins:repository-azure:qa:microsoft-azure-storage:namingConventions', task ':plugins:repository-azure:qa:microsoft-azure-storage:thirdPartyAudit', task ':plugins:repository-azure:qa:microsoft-azure-storage:precommit', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestCluster#init', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestCluster#wait', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestRunner', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTestCluster#stop', task ':plugins:repository-azure:qa:microsoft-azure-storage:azureStorageFixture#stop', task ':plugins:repository-azure:qa:microsoft-azure-storage:integTest', task ':plugins:repository-azure:qa:microsoft-azure-storage:check', task ':plugins:repository-azure:check', task ':plugins:repository-azure:build', task ':plugins:repository-gcs:generateNotice', task ':plugins:repository-gcs:compileJava', task ':plugins:repository-gcs:processResources', task ':plugins:repository-gcs:classes', task ':plugins:repository-gcs:jar', task ':plugins:repository-gcs:copyPluginPropertiesTemplate', task ':plugins:repository-gcs:pluginProperties', task ':plugins:repository-gcs:bundlePlugin', task ':plugins:repository-gcs:generatePomFileForZipPublication', task ':plugins:repository-gcs:generatePomFileForZipRealPublication', task ':plugins:repository-gcs:javadoc', task ':plugins:repository-gcs:javadocJar', task ':plugins:repository-gcs:sourcesJar', task ':plugins:repository-gcs:assemble', task ':plugins:repository-gcs:copyCheckstyleConf', task ':plugins:repository-gcs:checkstyleMain', task ':plugins:repository-gcs:compileTestJava', task ':plugins:repository-gcs:processTestResources', task ':plugins:repository-gcs:testClasses', task ':plugins:repository-gcs:checkstyleTest', task ':plugins:repository-gcs:forbiddenApisMain', task ':plugins:repository-gcs:forbiddenApisTest', task ':plugins:repository-gcs:forbiddenApis', task ':plugins:repository-gcs:integTestCluster#prepareCluster.cleanShared', task ':plugins:repository-gcs:integTestCluster#clean', task ':plugins:repository-gcs:integTestCluster#createCwd', task ':plugins:repository-gcs:integTestCluster#checkPrevious', task ':plugins:repository-gcs:integTestCluster#stopPrevious', task ':plugins:repository-gcs:integTestCluster#extract', task ':plugins:repository-gcs:integTestCluster#configure', task ':plugins:repository-gcs:integTestCluster#copyPlugins', task ':plugins:repository-gcs:integTestCluster#installRepositoryGcsPlugin', task ':plugins:repository-gcs:integTestCluster#start', task ':plugins:repository-gcs:copyRestSpec', task ':plugins:repository-gcs:checkstyle', task ':plugins:repository-gcs:dependencyLicenses', task ':plugins:repository-gcs:filepermissions', task ':plugins:repository-gcs:forbiddenPatterns', task ':plugins:repository-gcs:jarHell', task ':plugins:repository-gcs:licenseHeaders', task ':plugins:repository-gcs:loggerUsageCheck', task ':plugins:repository-gcs:namingConventions', task ':plugins:repository-gcs:thirdPartyAudit', task ':plugins:repository-gcs:precommit', task ':plugins:repository-gcs:test', task ':plugins:repository-gcs:integTestCluster#init', task ':plugins:repository-gcs:integTestCluster#wait', task ':plugins:repository-gcs:integTestRunner', task ':plugins:repository-gcs:integTestCluster#stop', task ':plugins:repository-gcs:integTest', task ':plugins:repository-gcs:extractJar', task ':plugins:repository-gcs:verifyJarLicense', task ':plugins:repository-gcs:verifyJarNotice', task ':plugins:repository-gcs:extractJavadocJar', task ':plugins:repository-gcs:verifyJavadocJarLicense', task ':plugins:repository-gcs:verifyJavadocJarNotice', task ':plugins:repository-gcs:extractSourcesJar', task ':plugins:repository-gcs:verifySourcesJarLicense', task ':plugins:repository-gcs:verifySourcesJarNotice', task ':plugins:repository-gcs:qa:google-cloud-storage:copyCheckstyleConf', task ':plugins:repository-gcs:qa:google-cloud-storage:compileTestJava', task ':plugins:repository-gcs:qa:google-cloud-storage:processTestResources', task ':plugins:repository-gcs:qa:google-cloud-storage:testClasses', task ':plugins:repository-gcs:qa:google-cloud-storage:checkstyleTest', task ':plugins:repository-gcs:qa:google-cloud-storage:forbiddenApisTest', task ':plugins:repository-gcs:qa:google-cloud-storage:forbiddenApis', task ':plugins:repository-gcs:qa:google-cloud-storage:googleCloudStorageFixture', task ':plugins:repository-gcs:qa:google-cloud-storage:createServiceAccountFile', task ':plugins:repository-gcs:qa:google-cloud-storage:integTestCluster#prepareCluster.cleanShared', task ':plugins:repository-gcs:qa:google-cloud-storage:integTestCluster#clean', task ':plugins:repository-gcs:qa:google-cloud-storage:integTestCluster#createCwd', task ':plugins:repository-gcs:qa:google-cloud-storage:integTestCluster#checkPrevious', task ':plugins:repository-gcs:qa:google-cloud-storage:integTestCluster#stopPrevious', task ':plugins:repository-gcs:qa:google-cloud-storage:integTestCluster#extract', task ':plugins:repository-gcs:qa:google-cloud-storage:integTestCluster#configure', task ':plugins:repository-gcs:qa:google-cloud-storage:integTestCluster#createKeystore', task ':plugins:repository-gcs:qa:google-cloud-storage:integTestCluster#addToKeystore#gcs.client.integration_test.credentials_file', task ':plugins:repository-gcs:qa:google-cloud-storage:integTestCluster#copyPlugins', task ':plugins:repository-gcs:qa:google-cloud-storage:integTestCluster#installRepositoryGcsPlugin', task ':plugins:repository-gcs:qa:google-cloud-storage:integTestCluster#start', task ':plugins:repository-gcs:qa:google-cloud-storage:copyRestSpec', task ':plugins:repository-gcs:qa:google-cloud-storage:checkstyle', task ':plugins:repository-gcs:qa:google-cloud-storage:filepermissions', task ':plugins:repository-gcs:qa:google-cloud-storage:forbiddenPatterns', task ':plugins:repository-gcs:qa:google-cloud-storage:jarHell', task ':plugins:repository-gcs:qa:google-cloud-storage:licenseHeaders', task ':plugins:repository-gcs:qa:google-cloud-storage:loggerUsageCheck', task ':plugins:repository-gcs:qa:google-cloud-storage:namingConventions', task ':plugins:repository-gcs:qa:google-cloud-storage:thirdPartyAudit', task ':plugins:repository-gcs:qa:google-cloud-storage:precommit', task ':plugins:repository-gcs:qa:google-cloud-storage:integTestCluster#init', task ':plugins:repository-gcs:qa:google-cloud-storage:integTestCluster#wait', task ':plugins:repository-gcs:qa:google-cloud-storage:integTestRunner', task ':plugins:repository-gcs:qa:google-cloud-storage:integTestCluster#stop', task ':plugins:repository-gcs:qa:google-cloud-storage:googleCloudStorageFixture#stop', task ':plugins:repository-gcs:qa:google-cloud-storage:integTest', task ':plugins:repository-gcs:qa:google-cloud-storage:check', task ':plugins:repository-gcs:check', task ':plugins:repository-gcs:build', task ':plugins:repository-hdfs:generateNotice', task ':plugins:repository-hdfs:compileJava', task ':plugins:repository-hdfs:processResources', task ':plugins:repository-hdfs:classes', task ':plugins:repository-hdfs:jar', task ':plugins:repository-hdfs:copyPluginPropertiesTemplate', task ':plugins:repository-hdfs:pluginProperties', task ':plugins:repository-hdfs:bundlePlugin', task ':plugins:repository-hdfs:generatePomFileForZipPublication', task ':plugins:repository-hdfs:generatePomFileForZipRealPublication', task ':plugins:repository-hdfs:javadoc', task ':plugins:repository-hdfs:javadocJar', task ':plugins:repository-hdfs:sourcesJar', task ':plugins:repository-hdfs:assemble', task ':plugins:repository-hdfs:copyCheckstyleConf', task ':plugins:repository-hdfs:checkstyleMain', task ':plugins:repository-hdfs:compileTestJava', task ':plugins:repository-hdfs:processTestResources', task ':plugins:repository-hdfs:testClasses', task ':plugins:repository-hdfs:checkstyleTest', task ':plugins:repository-hdfs:forbiddenApisMain', task ':plugins:repository-hdfs:forbiddenApisTest', task ':plugins:repository-hdfs:forbiddenApis', task ':plugins:repository-hdfs:integTestCluster#prepareCluster.cleanShared', task ':plugins:repository-hdfs:integTestCluster#clean', task ':plugins:repository-hdfs:integTestCluster#createCwd', task ':plugins:repository-hdfs:integTestCluster#checkPrevious', task ':plugins:repository-hdfs:integTestCluster#stopPrevious', task ':plugins:repository-hdfs:integTestCluster#extract', task ':plugins:repository-hdfs:integTestCluster#configure', task ':plugins:repository-hdfs:integTestCluster#copyPlugins', task ':plugins:repository-hdfs:integTestCluster#installRepositoryHdfsPlugin', task ':plugins:repository-hdfs:integTestCluster#start', task ':plugins:repository-hdfs:copyRestSpec', task ':plugins:repository-hdfs:checkstyle', task ':plugins:repository-hdfs:dependencyLicenses', task ':plugins:repository-hdfs:filepermissions', task ':plugins:repository-hdfs:forbiddenPatterns', task ':plugins:repository-hdfs:jarHell', task ':plugins:repository-hdfs:licenseHeaders', task ':plugins:repository-hdfs:loggerUsageCheck', task ':plugins:repository-hdfs:namingConventions', task ':plugins:repository-hdfs:thirdPartyAudit', task ':plugins:repository-hdfs:precommit', task ':test:fixtures:hdfs-fixture:compileJava', task ':test:fixtures:hdfs-fixture:processResources', task ':test:fixtures:hdfs-fixture:classes', task ':test:fixtures:hdfs-fixture:copyCheckstyleConf', task ':test:fixtures:hdfs-fixture:checkstyleMain', task ':test:fixtures:hdfs-fixture:compileTestJava', task ':test:fixtures:hdfs-fixture:processTestResources', task ':test:fixtures:hdfs-fixture:testClasses', task ':test:fixtures:hdfs-fixture:checkstyleTest', task ':test:fixtures:hdfs-fixture:checkstyle', task ':test:fixtures:hdfs-fixture:dependencyLicenses', task ':test:fixtures:hdfs-fixture:filepermissions', task ':test:fixtures:hdfs-fixture:forbiddenApisMain', task ':test:fixtures:hdfs-fixture:forbiddenApisTest', task ':test:fixtures:hdfs-fixture:forbiddenApis', task ':test:fixtures:hdfs-fixture:forbiddenPatterns', task ':test:fixtures:hdfs-fixture:jarHell', task ':test:fixtures:hdfs-fixture:licenseHeaders', task ':test:fixtures:hdfs-fixture:loggerUsageCheck', task ':test:fixtures:hdfs-fixture:namingConventions', task ':test:fixtures:hdfs-fixture:thirdPartyAudit', task ':test:fixtures:hdfs-fixture:precommit', task ':test:fixtures:hdfs-fixture:test', task ':plugins:repository-hdfs:test', task ':plugins:repository-hdfs:integTestCluster#init', task ':plugins:repository-hdfs:integTestCluster#wait', task ':plugins:repository-hdfs:integTestRunner', task ':plugins:repository-hdfs:integTestCluster#stop', task ':plugins:repository-hdfs:integTest', task ':plugins:repository-hdfs:extractJar', task ':plugins:repository-hdfs:verifyJarLicense', task ':plugins:repository-hdfs:verifyJarNotice', task ':plugins:repository-hdfs:extractJavadocJar', task ':plugins:repository-hdfs:verifyJavadocJarLicense', task ':plugins:repository-hdfs:verifyJavadocJarNotice', task ':plugins:repository-hdfs:extractSourcesJar', task ':plugins:repository-hdfs:verifySourcesJarLicense', task ':plugins:repository-hdfs:verifySourcesJarNotice', task ':plugins:repository-hdfs:check', task ':plugins:repository-hdfs:build', task ':plugins:repository-s3:generateNotice', task ':plugins:repository-s3:compileJava', task ':plugins:repository-s3:processResources', task ':plugins:repository-s3:classes', task ':plugins:repository-s3:jar', task ':plugins:repository-s3:copyPluginPropertiesTemplate', task ':plugins:repository-s3:pluginProperties', task ':plugins:repository-s3:bundlePlugin', task ':plugins:repository-s3:generatePomFileForZipPublication', task ':plugins:repository-s3:generatePomFileForZipRealPublication', task ':plugins:repository-s3:javadoc', task ':plugins:repository-s3:javadocJar', task ':plugins:repository-s3:sourcesJar', task ':plugins:repository-s3:assemble', task ':plugins:repository-s3:copyCheckstyleConf', task ':plugins:repository-s3:checkstyleMain', task ':plugins:repository-s3:compileTestJava', task ':plugins:repository-s3:processTestResources', task ':plugins:repository-s3:testClasses', task ':plugins:repository-s3:checkstyleTest', task ':plugins:repository-s3:forbiddenApisMain', task ':plugins:repository-s3:forbiddenApisTest', task ':plugins:repository-s3:forbiddenApis', task ':plugins:repository-s3:integTestCluster#prepareCluster.cleanShared', task ':plugins:repository-s3:integTestCluster#clean', task ':plugins:repository-s3:integTestCluster#createCwd', task ':plugins:repository-s3:integTestCluster#checkPrevious', task ':plugins:repository-s3:integTestCluster#stopPrevious', task ':plugins:repository-s3:integTestCluster#extract', task ':plugins:repository-s3:integTestCluster#configure', task ':plugins:repository-s3:integTestCluster#createKeystore', task ':plugins:repository-s3:integTestCluster#addToKeystore#s3.client.integration_test.access_key', task ':plugins:repository-s3:integTestCluster#addToKeystore#s3.client.integration_test.secret_key', task ':plugins:repository-s3:integTestCluster#copyPlugins', task ':plugins:repository-s3:integTestCluster#installRepositoryS3Plugin', task ':plugins:repository-s3:integTestCluster#start', task ':plugins:repository-s3:copyRestSpec', task ':plugins:repository-s3:checkstyle', task ':plugins:repository-s3:dependencyLicenses', task ':plugins:repository-s3:filepermissions', task ':plugins:repository-s3:forbiddenPatterns', task ':plugins:repository-s3:jarHell', task ':plugins:repository-s3:licenseHeaders', task ':plugins:repository-s3:loggerUsageCheck', task ':plugins:repository-s3:namingConventions', task ':plugins:repository-s3:thirdPartyAudit', task ':plugins:repository-s3:precommit', task ':plugins:repository-s3:testRepositoryCreds', task ':plugins:repository-s3:test', task ':plugins:repository-s3:integTestCluster#init', task ':plugins:repository-s3:integTestCluster#wait', task ':plugins:repository-s3:integTestRunner', task ':plugins:repository-s3:integTestCluster#stop', task ':plugins:repository-s3:integTest', task ':plugins:repository-s3:extractJar', task ':plugins:repository-s3:verifyJarLicense', task ':plugins:repository-s3:verifyJarNotice', task ':plugins:repository-s3:extractJavadocJar', task ':plugins:repository-s3:verifyJavadocJarLicense', task ':plugins:repository-s3:verifyJavadocJarNotice', task ':plugins:repository-s3:extractSourcesJar', task ':plugins:repository-s3:verifySourcesJarLicense', task ':plugins:repository-s3:verifySourcesJarNotice', task ':plugins:repository-s3:qa:amazon-s3:copyCheckstyleConf', task ':plugins:repository-s3:qa:amazon-s3:compileTestJava', task ':plugins:repository-s3:qa:amazon-s3:processTestResources', task ':plugins:repository-s3:qa:amazon-s3:testClasses', task ':plugins:repository-s3:qa:amazon-s3:checkstyleTest', task ':plugins:repository-s3:qa:amazon-s3:forbiddenApisTest', task ':plugins:repository-s3:qa:amazon-s3:forbiddenApis', task ':plugins:repository-s3:qa:amazon-s3:s3Fixture', task ':plugins:repository-s3:qa:amazon-s3:integTestCluster#prepareCluster.cleanShared', task ':plugins:repository-s3:qa:amazon-s3:integTestCluster#clean', task ':plugins:repository-s3:qa:amazon-s3:integTestCluster#createCwd', task ':plugins:repository-s3:qa:amazon-s3:integTestCluster#checkPrevious', task ':plugins:repository-s3:qa:amazon-s3:integTestCluster#stopPrevious', task ':plugins:repository-s3:qa:amazon-s3:integTestCluster#extract', task ':plugins:repository-s3:qa:amazon-s3:integTestCluster#configure', task ':plugins:repository-s3:qa:amazon-s3:integTestCluster#createKeystore', task ':plugins:repository-s3:qa:amazon-s3:integTestCluster#addToKeystore#s3.client.integration_test.access_key', task ':plugins:repository-s3:qa:amazon-s3:integTestCluster#addToKeystore#s3.client.integration_test.secret_key', task ':plugins:repository-s3:qa:amazon-s3:integTestCluster#copyPlugins', task ':plugins:repository-s3:qa:amazon-s3:integTestCluster#installRepositoryS3Plugin', task ':plugins:repository-s3:qa:amazon-s3:integTestCluster#start', task ':plugins:repository-s3:qa:amazon-s3:copyRestSpec', task ':plugins:repository-s3:qa:amazon-s3:checkstyle', task ':plugins:repository-s3:qa:amazon-s3:filepermissions', task ':plugins:repository-s3:qa:amazon-s3:forbiddenPatterns', task ':plugins:repository-s3:qa:amazon-s3:jarHell', task ':plugins:repository-s3:qa:amazon-s3:licenseHeaders', task ':plugins:repository-s3:qa:amazon-s3:loggerUsageCheck', task ':plugins:repository-s3:qa:amazon-s3:namingConventions', task ':plugins:repository-s3:qa:amazon-s3:thirdPartyAudit', task ':plugins:repository-s3:qa:amazon-s3:precommit', task ':plugins:repository-s3:qa:amazon-s3:integTestCluster#init', task ':plugins:repository-s3:qa:amazon-s3:integTestCluster#wait', task ':plugins:repository-s3:qa:amazon-s3:integTestRunner', task ':plugins:repository-s3:qa:amazon-s3:s3Fixture#stop', task ':plugins:repository-s3:qa:amazon-s3:integTestCluster#stop', task ':plugins:repository-s3:qa:amazon-s3:integTest', task ':plugins:repository-s3:qa:amazon-s3:check', task ':plugins:repository-s3:check', task ':plugins:repository-s3:build', task ':plugins:store-smb:generatePomFileForZipPublication', task ':plugins:store-smb:generatePomFileForZipRealPublication', task ':plugins:store-smb:javadoc', task ':plugins:store-smb:javadocJar', task ':plugins:store-smb:sourcesJar', task ':plugins:store-smb:assemble', task ':plugins:store-smb:extractJar', task ':plugins:store-smb:verifyJarLicense', task ':plugins:store-smb:verifyJarNotice', task ':plugins:store-smb:extractJavadocJar', task ':plugins:store-smb:verifyJavadocJarLicense', task ':plugins:store-smb:verifyJavadocJarNotice', task ':plugins:store-smb:extractSourcesJar', task ':plugins:store-smb:verifySourcesJarLicense', task ':plugins:store-smb:verifySourcesJarNotice', task ':plugins:store-smb:check', task ':plugins:store-smb:build', task ':plugins:transport-nio:generatePomFileForZipPublication', task ':plugins:transport-nio:generatePomFileForZipRealPublication', task ':plugins:transport-nio:javadoc', task ':plugins:transport-nio:javadocJar', task ':plugins:transport-nio:sourcesJar', task ':plugins:transport-nio:assemble', task ':plugins:transport-nio:extractJar', task ':plugins:transport-nio:verifyJarLicense', task ':plugins:transport-nio:verifyJarNotice', task ':plugins:transport-nio:extractJavadocJar', task ':plugins:transport-nio:verifyJavadocJarLicense', task ':plugins:transport-nio:verifyJavadocJarNotice', task ':plugins:transport-nio:extractSourcesJar', task ':plugins:transport-nio:verifySourcesJarLicense', task ':plugins:transport-nio:verifySourcesJarNotice', task ':plugins:transport-nio:check', task ':plugins:transport-nio:build', task ':qa:ccs-unavailable-clusters:copyCheckstyleConf', task ':qa:ccs-unavailable-clusters:compileTestJava', task ':qa:ccs-unavailable-clusters:processTestResources', task ':qa:ccs-unavailable-clusters:testClasses', task ':qa:ccs-unavailable-clusters:checkstyleTest', task ':qa:ccs-unavailable-clusters:forbiddenApisTest', task ':qa:ccs-unavailable-clusters:forbiddenApis', task ':qa:ccs-unavailable-clusters:integTestCluster#prepareCluster.cleanShared', task ':qa:ccs-unavailable-clusters:integTestCluster#clean', task ':qa:ccs-unavailable-clusters:integTestCluster#createCwd', task ':qa:ccs-unavailable-clusters:integTestCluster#checkPrevious', task ':qa:ccs-unavailable-clusters:integTestCluster#stopPrevious', task ':qa:ccs-unavailable-clusters:integTestCluster#extract', task ':qa:ccs-unavailable-clusters:integTestCluster#configure', task ':qa:ccs-unavailable-clusters:integTestCluster#start', task ':qa:ccs-unavailable-clusters:copyRestSpec', task ':qa:ccs-unavailable-clusters:checkstyle', task ':qa:ccs-unavailable-clusters:filepermissions', task ':qa:ccs-unavailable-clusters:forbiddenPatterns', task ':qa:ccs-unavailable-clusters:jarHell', task ':qa:ccs-unavailable-clusters:licenseHeaders', task ':qa:ccs-unavailable-clusters:loggerUsageCheck', task ':qa:ccs-unavailable-clusters:namingConventions', task ':qa:ccs-unavailable-clusters:thirdPartyAudit', task ':qa:ccs-unavailable-clusters:precommit', task ':qa:ccs-unavailable-clusters:integTestCluster#init', task ':qa:ccs-unavailable-clusters:integTestCluster#wait', task ':qa:ccs-unavailable-clusters:integTestRunner', task ':qa:ccs-unavailable-clusters:integTestCluster#stop', task ':qa:ccs-unavailable-clusters:integTest', task ':qa:ccs-unavailable-clusters:check', task ':qa:ccs-unavailable-clusters:build', task ':qa:die-with-dignity:compileJava', task ':qa:die-with-dignity:processResources', task ':qa:die-with-dignity:classes', task ':qa:die-with-dignity:copyCheckstyleConf', task ':qa:die-with-dignity:checkstyleMain', task ':qa:die-with-dignity:copyPluginPropertiesTemplate', task ':qa:die-with-dignity:pluginProperties', task ':qa:die-with-dignity:compileTestJava', task ':qa:die-with-dignity:processTestResources', task ':qa:die-with-dignity:testClasses', task ':qa:die-with-dignity:checkstyleTest', task ':qa:die-with-dignity:forbiddenApisMain', task ':qa:die-with-dignity:forbiddenApisTest', task ':qa:die-with-dignity:forbiddenApis', task ':qa:die-with-dignity:jar', task ':qa:die-with-dignity:bundlePlugin', task ':qa:die-with-dignity:integTestCluster#prepareCluster.cleanShared', task ':qa:die-with-dignity:integTestCluster#clean', task ':qa:die-with-dignity:integTestCluster#createCwd', task ':qa:die-with-dignity:integTestCluster#checkPrevious', task ':qa:die-with-dignity:integTestCluster#stopPrevious', task ':qa:die-with-dignity:integTestCluster#extract', task ':qa:die-with-dignity:integTestCluster#configure', task ':qa:die-with-dignity:integTestCluster#copyPlugins', task ':qa:die-with-dignity:integTestCluster#installDieWithDignityPlugin', task ':qa:die-with-dignity:integTestCluster#start', task ':qa:die-with-dignity:copyRestSpec', task ':qa:die-with-dignity:checkstyle', task ':qa:die-with-dignity:dependencyLicenses', task ':qa:die-with-dignity:filepermissions', task ':qa:die-with-dignity:forbiddenPatterns', task ':qa:die-with-dignity:jarHell', task ':qa:die-with-dignity:licenseHeaders', task ':qa:die-with-dignity:loggerUsageCheck', task ':qa:die-with-dignity:namingConventions', task ':qa:die-with-dignity:thirdPartyAudit', task ':qa:die-with-dignity:precommit', task ':qa:die-with-dignity:test', task ':qa:die-with-dignity:integTestCluster#init', task ':qa:die-with-dignity:integTestCluster#wait', task ':qa:die-with-dignity:integTestRunner', task ':qa:die-with-dignity:integTestCluster#stop', task ':qa:die-with-dignity:integTest', task ':qa:die-with-dignity:extractJar', task ':qa:die-with-dignity:verifyJarLicense', task ':qa:die-with-dignity:verifyJarNotice', task ':qa:die-with-dignity:javadoc', task ':qa:die-with-dignity:javadocJar', task ':qa:die-with-dignity:extractJavadocJar', task ':qa:die-with-dignity:verifyJavadocJarLicense', task ':qa:die-with-dignity:verifyJavadocJarNotice', task ':qa:die-with-dignity:sourcesJar', task ':qa:die-with-dignity:extractSourcesJar', task ':qa:die-with-dignity:verifySourcesJarLicense', task ':qa:die-with-dignity:verifySourcesJarNotice', task ':qa:die-with-dignity:check', task ':qa:die-with-dignity:build', task ':qa:evil-tests:copyCheckstyleConf', task ':qa:evil-tests:compileTestJava', task ':qa:evil-tests:processTestResources', task ':qa:evil-tests:testClasses', task ':qa:evil-tests:checkstyleTest', task ':qa:evil-tests:forbiddenApisTest', task ':qa:evil-tests:forbiddenApis', task ':qa:evil-tests:checkstyle', task ':qa:evil-tests:filepermissions', task ':qa:evil-tests:forbiddenPatterns', task ':qa:evil-tests:jarHell', task ':qa:evil-tests:licenseHeaders', task ':qa:evil-tests:loggerUsageCheck', task ':qa:evil-tests:namingConventions', task ':qa:evil-tests:thirdPartyAudit', task ':qa:evil-tests:precommit', task ':qa:evil-tests:test', task ':qa:evil-tests:check', task ':qa:evil-tests:build', task ':qa:full-cluster-restart:copyCheckstyleConf', task ':qa:full-cluster-restart:compileTestJava', task ':qa:full-cluster-restart:processTestResources', task ':qa:full-cluster-restart:testClasses', task ':qa:full-cluster-restart:checkstyleTest', task ':qa:full-cluster-restart:forbiddenApisTest', task ':qa:full-cluster-restart:forbiddenApis', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#prepareCluster.cleanShared', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.clean', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.createCwd', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.checkPrevious', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.stopPrevious', task ':distribution:bwc:next-bugfix-snapshot:createClone', task ':distribution:bwc:next-bugfix-snapshot:findRemote', task ':distribution:bwc:next-bugfix-snapshot:addRemote', task ':distribution:bwc:next-bugfix-snapshot:fetchLatest', task ':distribution:bwc:next-bugfix-snapshot:checkoutBwcBranch', task ':distribution:bwc:next-bugfix-snapshot:writeBuildMetadata', task ':distribution:bwc:next-bugfix-snapshot:buildBwcVersion', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.extract', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.configure', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.start', task ':qa:full-cluster-restart:copyRestSpec', task ':qa:full-cluster-restart:checkstyle', task ':qa:full-cluster-restart:filepermissions', task ':qa:full-cluster-restart:forbiddenPatterns', task ':qa:full-cluster-restart:jarHell', task ':qa:full-cluster-restart:licenseHeaders', task ':qa:full-cluster-restart:loggerUsageCheck', task ':qa:full-cluster-restart:namingConventions', task ':qa:full-cluster-restart:thirdPartyAudit', task ':qa:full-cluster-restart:precommit', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#init', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.clean', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.createCwd', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.checkPrevious', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.stopPrevious', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.extract', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.configure', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.start', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#wait', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestRunner', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.stop', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.stop', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#oldClusterTest', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.clean', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.createCwd', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.checkPrevious', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.stopPrevious', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.extract', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.configure', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.start', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#init', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.clean', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.createCwd', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.checkPrevious', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.stopPrevious', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.extract', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.configure', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.start', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#wait', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestRunner', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.stop', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.stop', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#upgradedClusterTest', task ':qa:full-cluster-restart:v6.2.5-SNAPSHOT#bwcTest', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#prepareCluster.cleanShared', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.clean', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.createCwd', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.checkPrevious', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.stopPrevious', task ':distribution:bwc:staged-minor-snapshot:createClone', task ':distribution:bwc:staged-minor-snapshot:findRemote', task ':distribution:bwc:staged-minor-snapshot:addRemote', task ':distribution:bwc:staged-minor-snapshot:fetchLatest', task ':distribution:bwc:staged-minor-snapshot:checkoutBwcBranch', task ':distribution:bwc:staged-minor-snapshot:writeBuildMetadata', task ':distribution:bwc:staged-minor-snapshot:buildBwcVersion', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.extract', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.configure', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.start', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#init', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.clean', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.createCwd', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.checkPrevious', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.stopPrevious', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.extract', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.configure', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.start', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#wait', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestRunner', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.stop', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.stop', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#oldClusterTest', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.clean', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.createCwd', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.checkPrevious', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.stopPrevious', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.extract', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.configure', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.start', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#init', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.clean', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.createCwd', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.checkPrevious', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.stopPrevious', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.extract', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.configure', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.start', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#wait', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestRunner', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.stop', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.stop', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#upgradedClusterTest', task ':qa:full-cluster-restart:v6.3.0-SNAPSHOT#bwcTest', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#prepareCluster.cleanShared', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.clean', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.createCwd', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.checkPrevious', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.stopPrevious', task ':distribution:bwc:next-minor-snapshot:createClone', task ':distribution:bwc:next-minor-snapshot:findRemote', task ':distribution:bwc:next-minor-snapshot:addRemote', task ':distribution:bwc:next-minor-snapshot:fetchLatest', task ':distribution:bwc:next-minor-snapshot:checkoutBwcBranch', task ':distribution:bwc:next-minor-snapshot:writeBuildMetadata', task ':distribution:bwc:next-minor-snapshot:buildBwcVersion', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.extract', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.configure', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.start', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#init', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.clean', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.createCwd', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.checkPrevious', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.stopPrevious', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.extract', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.configure', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.start', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#wait', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestRunner', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.stop', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.stop', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#oldClusterTest', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.clean', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.createCwd', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.checkPrevious', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.stopPrevious', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.extract', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.configure', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.start', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#init', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.clean', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.createCwd', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.checkPrevious', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.stopPrevious', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.extract', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.configure', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.start', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#wait', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestRunner', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.stop', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.stop', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#upgradedClusterTest', task ':qa:full-cluster-restart:v6.4.0-SNAPSHOT#bwcTest', task ':qa:full-cluster-restart:integTest', task ':qa:full-cluster-restart:test', task ':qa:full-cluster-restart:check', task ':qa:full-cluster-restart:build', task ':qa:mixed-cluster:copyCheckstyleConf', task ':qa:mixed-cluster:compileTestJava', task ':qa:mixed-cluster:processTestResources', task ':qa:mixed-cluster:testClasses', task ':qa:mixed-cluster:checkstyleTest', task ':qa:mixed-cluster:forbiddenApisTest', task ':qa:mixed-cluster:forbiddenApis', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#prepareCluster.cleanShared', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node0.clean', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node0.createCwd', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node0.checkPrevious', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node0.stopPrevious', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node0.extract', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node0.configure', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node0.start', task ':qa:mixed-cluster:copyRestSpec', task ':qa:mixed-cluster:checkstyle', task ':qa:mixed-cluster:filepermissions', task ':qa:mixed-cluster:forbiddenPatterns', task ':qa:mixed-cluster:jarHell', task ':qa:mixed-cluster:licenseHeaders', task ':qa:mixed-cluster:loggerUsageCheck', task ':qa:mixed-cluster:namingConventions', task ':qa:mixed-cluster:thirdPartyAudit', task ':qa:mixed-cluster:precommit', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#init', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node1.clean', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node1.createCwd', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node1.checkPrevious', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node1.stopPrevious', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node1.extract', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node1.configure', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node1.start', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node2.clean', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node2.createCwd', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node2.checkPrevious', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node2.stopPrevious', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node2.extract', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node2.configure', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node2.start', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node3.clean', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node3.createCwd', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node3.checkPrevious', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node3.stopPrevious', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node3.extract', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node3.configure', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node3.start', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#wait', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestRunner', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node3.stop', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node2.stop', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node1.stop', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTestCluster#node0.stop', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#mixedClusterTest', task ':qa:mixed-cluster:v6.4.0-SNAPSHOT#bwcTest', task ':qa:mixed-cluster:integTest', task ':qa:mixed-cluster:test', task ':qa:mixed-cluster:check', task ':qa:mixed-cluster:build', task ':qa:multi-cluster-search:copyCheckstyleConf', task ':qa:multi-cluster-search:compileTestJava', task ':qa:multi-cluster-search:processTestResources', task ':qa:multi-cluster-search:testClasses', task ':qa:multi-cluster-search:checkstyleTest', task ':qa:multi-cluster-search:forbiddenApisTest', task ':qa:multi-cluster-search:forbiddenApis', task ':qa:multi-cluster-search:copyRestSpec', task ':qa:multi-cluster-search:checkstyle', task ':qa:multi-cluster-search:filepermissions', task ':qa:multi-cluster-search:forbiddenPatterns', task ':qa:multi-cluster-search:jarHell', task ':qa:multi-cluster-search:licenseHeaders', task ':qa:multi-cluster-search:loggerUsageCheck', task ':qa:multi-cluster-search:namingConventions', task ':qa:multi-cluster-search:thirdPartyAudit', task ':qa:multi-cluster-search:precommit', task ':qa:multi-cluster-search:remoteClusterTestCluster#init', task ':qa:multi-cluster-search:remoteClusterTestCluster#prepareCluster.cleanShared', task ':qa:multi-cluster-search:remoteClusterTestCluster#node0.clean', task ':qa:multi-cluster-search:remoteClusterTestCluster#node0.createCwd', task ':qa:multi-cluster-search:remoteClusterTestCluster#node0.checkPrevious', task ':qa:multi-cluster-search:remoteClusterTestCluster#node0.stopPrevious', task ':qa:multi-cluster-search:remoteClusterTestCluster#node0.extract', task ':qa:multi-cluster-search:remoteClusterTestCluster#node0.configure', task ':qa:multi-cluster-search:remoteClusterTestCluster#node0.start', task ':qa:multi-cluster-search:remoteClusterTestCluster#node1.clean', task ':qa:multi-cluster-search:remoteClusterTestCluster#node1.createCwd', task ':qa:multi-cluster-search:remoteClusterTestCluster#node1.checkPrevious', task ':qa:multi-cluster-search:remoteClusterTestCluster#node1.stopPrevious', task ':qa:multi-cluster-search:remoteClusterTestCluster#node1.extract', task ':qa:multi-cluster-search:remoteClusterTestCluster#node1.configure', task ':qa:multi-cluster-search:remoteClusterTestCluster#node1.start', task ':qa:multi-cluster-search:remoteClusterTestCluster#wait', task ':qa:multi-cluster-search:remoteClusterTestRunner', task ':qa:multi-cluster-search:mixedClusterTestCluster#prepareCluster.cleanShared', task ':qa:multi-cluster-search:mixedClusterTestCluster#clean', task ':qa:multi-cluster-search:mixedClusterTestCluster#createCwd', task ':qa:multi-cluster-search:mixedClusterTestCluster#checkPrevious', task ':qa:multi-cluster-search:mixedClusterTestCluster#stopPrevious', task ':qa:multi-cluster-search:mixedClusterTestCluster#extract', task ':qa:multi-cluster-search:mixedClusterTestCluster#configure', task ':qa:multi-cluster-search:mixedClusterTestCluster#start', task ':qa:multi-cluster-search:mixedClusterTestCluster#init', task ':qa:multi-cluster-search:mixedClusterTestCluster#wait', task ':qa:multi-cluster-search:mixedClusterTestRunner', task ':qa:multi-cluster-search:remoteClusterTestCluster#node0.stop', task ':qa:multi-cluster-search:remoteClusterTestCluster#node1.stop', task ':qa:multi-cluster-search:mixedClusterTestCluster#stop', task ':qa:multi-cluster-search:mixedClusterTest', task ':qa:multi-cluster-search:integTest', task ':qa:multi-cluster-search:test', task ':qa:multi-cluster-search:check', task ':qa:multi-cluster-search:build', task ':qa:no-bootstrap-tests:copyCheckstyleConf', task ':qa:no-bootstrap-tests:compileTestJava', task ':qa:no-bootstrap-tests:processTestResources', task ':qa:no-bootstrap-tests:testClasses', task ':qa:no-bootstrap-tests:checkstyleTest', task ':qa:no-bootstrap-tests:forbiddenApisTest', task ':qa:no-bootstrap-tests:forbiddenApis', task ':qa:no-bootstrap-tests:checkstyle', task ':qa:no-bootstrap-tests:filepermissions', task ':qa:no-bootstrap-tests:forbiddenPatterns', task ':qa:no-bootstrap-tests:jarHell', task ':qa:no-bootstrap-tests:licenseHeaders', task ':qa:no-bootstrap-tests:loggerUsageCheck', task ':qa:no-bootstrap-tests:namingConventions', task ':qa:no-bootstrap-tests:thirdPartyAudit', task ':qa:no-bootstrap-tests:precommit', task ':qa:no-bootstrap-tests:test', task ':qa:no-bootstrap-tests:check', task ':qa:no-bootstrap-tests:build', task ':qa:query-builder-bwc:copyCheckstyleConf', task ':qa:query-builder-bwc:compileTestJava', task ':qa:query-builder-bwc:processTestResources', task ':qa:query-builder-bwc:testClasses', task ':qa:query-builder-bwc:checkstyleTest', task ':qa:query-builder-bwc:forbiddenApisTest', task ':qa:query-builder-bwc:forbiddenApis', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#oldQueryBuilderTestCluster#prepareCluster.cleanShared', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#oldQueryBuilderTestCluster#clean', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#oldQueryBuilderTestCluster#createCwd', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#oldQueryBuilderTestCluster#checkPrevious', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#oldQueryBuilderTestCluster#stopPrevious', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#oldQueryBuilderTestCluster#extract', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#oldQueryBuilderTestCluster#configure', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#oldQueryBuilderTestCluster#start', task ':qa:query-builder-bwc:copyRestSpec', task ':qa:query-builder-bwc:checkstyle', task ':qa:query-builder-bwc:filepermissions', task ':qa:query-builder-bwc:forbiddenPatterns', task ':qa:query-builder-bwc:jarHell', task ':qa:query-builder-bwc:licenseHeaders', task ':qa:query-builder-bwc:loggerUsageCheck', task ':qa:query-builder-bwc:namingConventions', task ':qa:query-builder-bwc:thirdPartyAudit', task ':qa:query-builder-bwc:precommit', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#oldQueryBuilderTestCluster#init', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#oldQueryBuilderTestCluster#wait', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#oldQueryBuilderTestRunner', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#oldQueryBuilderTestCluster#stop', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#oldQueryBuilderTest', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#upgradedQueryBuilderTestCluster#clean', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#upgradedQueryBuilderTestCluster#createCwd', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#upgradedQueryBuilderTestCluster#checkPrevious', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#upgradedQueryBuilderTestCluster#stopPrevious', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#upgradedQueryBuilderTestCluster#extract', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#upgradedQueryBuilderTestCluster#configure', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#upgradedQueryBuilderTestCluster#start', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#upgradedQueryBuilderTestCluster#init', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#upgradedQueryBuilderTestCluster#wait', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#upgradedQueryBuilderTestRunner', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#upgradedQueryBuilderTestCluster#stop', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#upgradedQueryBuilderTest', task ':qa:query-builder-bwc:v6.4.0-SNAPSHOT#bwcTest', task ':qa:query-builder-bwc:integTest', task ':qa:query-builder-bwc:test', task ':qa:query-builder-bwc:check', task ':qa:query-builder-bwc:build', task ':qa:rolling-upgrade:copyCheckstyleConf', task ':qa:rolling-upgrade:compileTestJava', task ':qa:rolling-upgrade:processTestResources', task ':qa:rolling-upgrade:testClasses', task ':qa:rolling-upgrade:checkstyleTest', task ':qa:rolling-upgrade:forbiddenApisTest', task ':qa:rolling-upgrade:forbiddenApis', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#prepareCluster.cleanShared', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.clean', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.createCwd', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.checkPrevious', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.stopPrevious', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.extract', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.configure', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.start', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.clean', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.createCwd', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.checkPrevious', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.stopPrevious', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.extract', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.configure', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.start', task ':qa:rolling-upgrade:copyRestSpec', task ':qa:rolling-upgrade:checkstyle', task ':qa:rolling-upgrade:filepermissions', task ':qa:rolling-upgrade:forbiddenPatterns', task ':qa:rolling-upgrade:jarHell', task ':qa:rolling-upgrade:licenseHeaders', task ':qa:rolling-upgrade:loggerUsageCheck', task ':qa:rolling-upgrade:namingConventions', task ':qa:rolling-upgrade:thirdPartyAudit', task ':qa:rolling-upgrade:precommit', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#init', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#wait', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestRunner', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.stop', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#mixedClusterTestCluster#prepareCluster.cleanShared', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#mixedClusterTestCluster#clean', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#mixedClusterTestCluster#createCwd', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#mixedClusterTestCluster#checkPrevious', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#mixedClusterTestCluster#stopPrevious', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#mixedClusterTestCluster#extract', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#mixedClusterTestCluster#configure', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#mixedClusterTestCluster#start', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#mixedClusterTestCluster#init', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#mixedClusterTestCluster#wait', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#mixedClusterTestRunner', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.stop', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#init', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#prepareCluster.cleanShared', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#clean', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#createCwd', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#checkPrevious', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#stopPrevious', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#extract', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#configure', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#start', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#wait', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#upgradedClusterTestRunner', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#stop', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#mixedClusterTestCluster#stop', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#upgradedClusterTest', task ':qa:rolling-upgrade:v6.4.0-SNAPSHOT#bwcTest', task ':qa:rolling-upgrade:integTest', task ':qa:rolling-upgrade:test', task ':qa:rolling-upgrade:check', task ':qa:rolling-upgrade:build', task ':qa:smoke-test-client:copyCheckstyleConf', task ':qa:smoke-test-client:compileTestJava', task ':qa:smoke-test-client:processTestResources', task ':qa:smoke-test-client:testClasses', task ':qa:smoke-test-client:checkstyleTest', task ':qa:smoke-test-client:forbiddenApisTest', task ':qa:smoke-test-client:forbiddenApis', task ':qa:smoke-test-client:singleNodeIntegTestCluster#prepareCluster.cleanShared', task ':qa:smoke-test-client:singleNodeIntegTestCluster#clean', task ':qa:smoke-test-client:singleNodeIntegTestCluster#createCwd', task ':qa:smoke-test-client:singleNodeIntegTestCluster#checkPrevious', task ':qa:smoke-test-client:singleNodeIntegTestCluster#stopPrevious', task ':qa:smoke-test-client:singleNodeIntegTestCluster#extract', task ':qa:smoke-test-client:singleNodeIntegTestCluster#configure', task ':qa:smoke-test-client:singleNodeIntegTestCluster#start', task ':qa:smoke-test-client:copyRestSpec', task ':qa:smoke-test-client:checkstyle', task ':qa:smoke-test-client:filepermissions', task ':qa:smoke-test-client:forbiddenPatterns', task ':qa:smoke-test-client:jarHell', task ':qa:smoke-test-client:licenseHeaders', task ':qa:smoke-test-client:loggerUsageCheck', task ':qa:smoke-test-client:namingConventions', task ':qa:smoke-test-client:thirdPartyAudit', task ':qa:smoke-test-client:precommit', task ':qa:smoke-test-client:singleNodeIntegTestCluster#init', task ':qa:smoke-test-client:singleNodeIntegTestCluster#wait', task ':qa:smoke-test-client:singleNodeIntegTestRunner', task ':qa:smoke-test-client:singleNodeIntegTestCluster#stop', task ':qa:smoke-test-client:integTestCluster#prepareCluster.cleanShared', task ':qa:smoke-test-client:integTestCluster#clean', task ':qa:smoke-test-client:integTestCluster#createCwd', task ':qa:smoke-test-client:integTestCluster#checkPrevious', task ':qa:smoke-test-client:integTestCluster#stopPrevious', task ':qa:smoke-test-client:integTestCluster#extract', task ':qa:smoke-test-client:integTestCluster#configure', task ':qa:smoke-test-client:integTestCluster#start', task ':qa:smoke-test-client:integTestCluster#init', task ':qa:smoke-test-client:integTestCluster#wait', task ':qa:smoke-test-client:integTestRunner', task ':qa:smoke-test-client:integTestCluster#stop', task ':qa:smoke-test-client:integTest', task ':qa:smoke-test-client:check', task ':qa:smoke-test-client:build', task ':qa:smoke-test-http:copyCheckstyleConf', task ':qa:smoke-test-http:copyTransportNetty4Metadata', task ':qa:smoke-test-http:compileTestJava', task ':qa:smoke-test-http:processTestResources', task ':qa:smoke-test-http:testClasses', task ':qa:smoke-test-http:checkstyleTest', task ':qa:smoke-test-http:forbiddenApisTest', task ':qa:smoke-test-http:forbiddenApis', task ':qa:smoke-test-http:integTestCluster#prepareCluster.cleanShared', task ':qa:smoke-test-http:integTestCluster#clean', task ':qa:smoke-test-http:integTestCluster#createCwd', task ':qa:smoke-test-http:integTestCluster#checkPrevious', task ':qa:smoke-test-http:integTestCluster#stopPrevious', task ':qa:smoke-test-http:integTestCluster#extract', task ':qa:smoke-test-http:integTestCluster#configure', task ':qa:smoke-test-http:integTestCluster#start', task ':qa:smoke-test-http:copyRestSpec', task ':qa:smoke-test-http:checkstyle', task ':qa:smoke-test-http:filepermissions', task ':qa:smoke-test-http:forbiddenPatterns', task ':qa:smoke-test-http:jarHell', task ':qa:smoke-test-http:licenseHeaders', task ':qa:smoke-test-http:loggerUsageCheck', task ':qa:smoke-test-http:namingConventions', task ':qa:smoke-test-http:thirdPartyAudit', task ':qa:smoke-test-http:precommit', task ':qa:smoke-test-http:integTestCluster#init', task ':qa:smoke-test-http:integTestCluster#wait', task ':qa:smoke-test-http:integTestRunner', task ':qa:smoke-test-http:integTestCluster#stop', task ':qa:smoke-test-http:integTest', task ':qa:smoke-test-http:check', task ':qa:smoke-test-http:build', task ':qa:smoke-test-ingest-disabled:copyCheckstyleConf', task ':qa:smoke-test-ingest-disabled:compileTestJava', task ':qa:smoke-test-ingest-disabled:processTestResources', task ':qa:smoke-test-ingest-disabled:testClasses', task ':qa:smoke-test-ingest-disabled:checkstyleTest', task ':qa:smoke-test-ingest-disabled:forbiddenApisTest', task ':qa:smoke-test-ingest-disabled:forbiddenApis', task ':qa:smoke-test-ingest-disabled:integTestCluster#prepareCluster.cleanShared', task ':qa:smoke-test-ingest-disabled:integTestCluster#clean', task ':qa:smoke-test-ingest-disabled:integTestCluster#createCwd', task ':qa:smoke-test-ingest-disabled:integTestCluster#checkPrevious', task ':qa:smoke-test-ingest-disabled:integTestCluster#stopPrevious', task ':qa:smoke-test-ingest-disabled:integTestCluster#extract', task ':qa:smoke-test-ingest-disabled:integTestCluster#configure', task ':qa:smoke-test-ingest-disabled:integTestCluster#start', task ':qa:smoke-test-ingest-disabled:copyRestSpec', task ':qa:smoke-test-ingest-disabled:checkstyle', task ':qa:smoke-test-ingest-disabled:filepermissions', task ':qa:smoke-test-ingest-disabled:forbiddenPatterns', task ':qa:smoke-test-ingest-disabled:jarHell', task ':qa:smoke-test-ingest-disabled:licenseHeaders', task ':qa:smoke-test-ingest-disabled:loggerUsageCheck', task ':qa:smoke-test-ingest-disabled:namingConventions', task ':qa:smoke-test-ingest-disabled:thirdPartyAudit', task ':qa:smoke-test-ingest-disabled:precommit', task ':qa:smoke-test-ingest-disabled:integTestCluster#init', task ':qa:smoke-test-ingest-disabled:integTestCluster#wait', task ':qa:smoke-test-ingest-disabled:integTestRunner', task ':qa:smoke-test-ingest-disabled:integTestCluster#stop', task ':qa:smoke-test-ingest-disabled:integTest', task ':qa:smoke-test-ingest-disabled:check', task ':qa:smoke-test-ingest-disabled:build', task ':qa:smoke-test-ingest-with-all-dependencies:copyCheckstyleConf', task ':qa:smoke-test-ingest-with-all-dependencies:compileTestJava', task ':qa:smoke-test-ingest-with-all-dependencies:processTestResources', task ':qa:smoke-test-ingest-with-all-dependencies:testClasses', task ':qa:smoke-test-ingest-with-all-dependencies:checkstyleTest', task ':qa:smoke-test-ingest-with-all-dependencies:forbiddenApisTest', task ':qa:smoke-test-ingest-with-all-dependencies:forbiddenApis', task ':qa:smoke-test-ingest-with-all-dependencies:integTestCluster#prepareCluster.cleanShared', task ':qa:smoke-test-ingest-with-all-dependencies:integTestCluster#clean', task ':qa:smoke-test-ingest-with-all-dependencies:integTestCluster#createCwd', task ':qa:smoke-test-ingest-with-all-dependencies:integTestCluster#checkPrevious', task ':qa:smoke-test-ingest-with-all-dependencies:integTestCluster#stopPrevious', task ':qa:smoke-test-ingest-with-all-dependencies:integTestCluster#extract', task ':qa:smoke-test-ingest-with-all-dependencies:integTestCluster#configure', task ':qa:smoke-test-ingest-with-all-dependencies:integTestCluster#copyPlugins', task ':qa:smoke-test-ingest-with-all-dependencies:integTestCluster#installIngestGeoipPlugin', task ':qa:smoke-test-ingest-with-all-dependencies:integTestCluster#start', task ':qa:smoke-test-ingest-with-all-dependencies:copyRestSpec', task ':qa:smoke-test-ingest-with-all-dependencies:checkstyle', task ':qa:smoke-test-ingest-with-all-dependencies:filepermissions', task ':qa:smoke-test-ingest-with-all-dependencies:forbiddenPatterns', task ':qa:smoke-test-ingest-with-all-dependencies:jarHell', task ':qa:smoke-test-ingest-with-all-dependencies:licenseHeaders', task ':qa:smoke-test-ingest-with-all-dependencies:loggerUsageCheck', task ':qa:smoke-test-ingest-with-all-dependencies:namingConventions', task ':qa:smoke-test-ingest-with-all-dependencies:thirdPartyAudit', task ':qa:smoke-test-ingest-with-all-dependencies:precommit', task ':qa:smoke-test-ingest-with-all-dependencies:integTestCluster#init', task ':qa:smoke-test-ingest-with-all-dependencies:integTestCluster#wait', task ':qa:smoke-test-ingest-with-all-dependencies:integTestRunner', task ':qa:smoke-test-ingest-with-all-dependencies:integTestCluster#stop', task ':qa:smoke-test-ingest-with-all-dependencies:integTest', task ':qa:smoke-test-ingest-with-all-dependencies:check', task ':qa:smoke-test-ingest-with-all-dependencies:build', task ':qa:smoke-test-multinode:copyCheckstyleConf', task ':qa:smoke-test-multinode:compileTestJava', task ':qa:smoke-test-multinode:processTestResources', task ':qa:smoke-test-multinode:testClasses', task ':qa:smoke-test-multinode:checkstyleTest', task ':qa:smoke-test-multinode:forbiddenApisTest', task ':qa:smoke-test-multinode:forbiddenApis', task ':qa:smoke-test-multinode:integTestCluster#prepareCluster.cleanShared', task ':qa:smoke-test-multinode:integTestCluster#node0.clean', task ':qa:smoke-test-multinode:integTestCluster#node0.createCwd', task ':qa:smoke-test-multinode:integTestCluster#node0.checkPrevious', task ':qa:smoke-test-multinode:integTestCluster#node0.stopPrevious', task ':qa:smoke-test-multinode:integTestCluster#node0.extract', task ':qa:smoke-test-multinode:integTestCluster#node0.configure', task ':qa:smoke-test-multinode:integTestCluster#node0.start', task ':qa:smoke-test-multinode:copyRestSpec', task ':qa:smoke-test-multinode:checkstyle', task ':qa:smoke-test-multinode:filepermissions', task ':qa:smoke-test-multinode:forbiddenPatterns', task ':qa:smoke-test-multinode:jarHell', task ':qa:smoke-test-multinode:licenseHeaders', task ':qa:smoke-test-multinode:loggerUsageCheck', task ':qa:smoke-test-multinode:namingConventions', task ':qa:smoke-test-multinode:thirdPartyAudit', task ':qa:smoke-test-multinode:precommit', task ':qa:smoke-test-multinode:integTestCluster#init', task ':qa:smoke-test-multinode:integTestCluster#node1.clean', task ':qa:smoke-test-multinode:integTestCluster#node1.createCwd', task ':qa:smoke-test-multinode:integTestCluster#node1.checkPrevious', task ':qa:smoke-test-multinode:integTestCluster#node1.stopPrevious', task ':qa:smoke-test-multinode:integTestCluster#node1.extract', task ':qa:smoke-test-multinode:integTestCluster#node1.configure', task ':qa:smoke-test-multinode:integTestCluster#node1.start', task ':qa:smoke-test-multinode:integTestCluster#wait', task ':qa:smoke-test-multinode:integTestRunner', task ':qa:smoke-test-multinode:integTestCluster#node1.stop', task ':qa:smoke-test-multinode:integTestCluster#node0.stop', task ':qa:smoke-test-multinode:integTest', task ':qa:smoke-test-multinode:check', task ':qa:smoke-test-multinode:build', task ':qa:smoke-test-plugins:copyCheckstyleConf', task ':qa:smoke-test-plugins:compileTestJava', task ':qa:smoke-test-plugins:processTestResources', task ':qa:smoke-test-plugins:testClasses', task ':qa:smoke-test-plugins:checkstyleTest', task ':qa:smoke-test-plugins:forbiddenApisTest', task ':qa:smoke-test-plugins:forbiddenApis', task ':qa:smoke-test-plugins:integTestCluster#prepareCluster.cleanShared', task ':qa:smoke-test-plugins:integTestCluster#clean', task ':qa:smoke-test-plugins:integTestCluster#createCwd', task ':qa:smoke-test-plugins:integTestCluster#checkPrevious', task ':qa:smoke-test-plugins:integTestCluster#stopPrevious', task ':qa:smoke-test-plugins:integTestCluster#extract', task ':qa:smoke-test-plugins:integTestCluster#configure', task ':qa:smoke-test-plugins:integTestCluster#copyPlugins', task ':qa:smoke-test-plugins:integTestCluster#installAnalysisIcuPlugin', task ':qa:smoke-test-plugins:integTestCluster#installAnalysisKuromojiPlugin', task ':qa:smoke-test-plugins:integTestCluster#installAnalysisNoriPlugin', task ':qa:smoke-test-plugins:integTestCluster#installAnalysisPhoneticPlugin', task ':qa:smoke-test-plugins:integTestCluster#installAnalysisSmartcnPlugin', task ':qa:smoke-test-plugins:integTestCluster#installAnalysisStempelPlugin', task ':qa:smoke-test-plugins:integTestCluster#installAnalysisUkrainianPlugin', task ':qa:smoke-test-plugins:integTestCluster#installDiscoveryAzureClassicPlugin', task ':qa:smoke-test-plugins:integTestCluster#installDiscoveryEc2Plugin', task ':qa:smoke-test-plugins:integTestCluster#installDiscoveryFilePlugin', task ':qa:smoke-test-plugins:integTestCluster#installDiscoveryGcePlugin', task ':qa:smoke-test-plugins:integTestCluster#installIngestAttachmentPlugin', task ':qa:smoke-test-plugins:integTestCluster#installIngestGeoipPlugin', task ':qa:smoke-test-plugins:integTestCluster#installIngestUserAgentPlugin', task ':qa:smoke-test-plugins:integTestCluster#installMapperMurmur3Plugin', task ':qa:smoke-test-plugins:integTestCluster#installMapperSizePlugin', task ':qa:smoke-test-plugins:integTestCluster#installRepositoryAzurePlugin', task ':qa:smoke-test-plugins:integTestCluster#installRepositoryGcsPlugin', task ':qa:smoke-test-plugins:integTestCluster#installRepositoryHdfsPlugin', task ':qa:smoke-test-plugins:integTestCluster#installRepositoryS3Plugin', task ':qa:smoke-test-plugins:integTestCluster#installStoreSmbPlugin', task ':qa:smoke-test-plugins:integTestCluster#installTransportNioPlugin', task ':qa:smoke-test-plugins:integTestCluster#start', task ':qa:smoke-test-plugins:copyRestSpec', task ':qa:smoke-test-plugins:checkstyle', task ':qa:smoke-test-plugins:filepermissions', task ':qa:smoke-test-plugins:forbiddenPatterns', task ':qa:smoke-test-plugins:jarHell', task ':qa:smoke-test-plugins:licenseHeaders', task ':qa:smoke-test-plugins:loggerUsageCheck', task ':qa:smoke-test-plugins:namingConventions', task ':qa:smoke-test-plugins:thirdPartyAudit', task ':qa:smoke-test-plugins:precommit', task ':qa:smoke-test-plugins:integTestCluster#init', task ':qa:smoke-test-plugins:integTestCluster#wait', task ':qa:smoke-test-plugins:integTestRunner', task ':qa:smoke-test-plugins:integTestCluster#stop', task ':qa:smoke-test-plugins:integTest', task ':qa:smoke-test-plugins:check', task ':qa:smoke-test-plugins:build', task ':qa:smoke-test-rank-eval-with-mustache:copyCheckstyleConf', task ':qa:smoke-test-rank-eval-with-mustache:compileTestJava', task ':qa:smoke-test-rank-eval-with-mustache:processTestResources', task ':qa:smoke-test-rank-eval-with-mustache:testClasses', task ':qa:smoke-test-rank-eval-with-mustache:checkstyleTest', task ':qa:smoke-test-rank-eval-with-mustache:forbiddenApisTest', task ':qa:smoke-test-rank-eval-with-mustache:forbiddenApis', task ':qa:smoke-test-rank-eval-with-mustache:integTestCluster#prepareCluster.cleanShared', task ':qa:smoke-test-rank-eval-with-mustache:integTestCluster#clean', task ':qa:smoke-test-rank-eval-with-mustache:integTestCluster#createCwd', task ':qa:smoke-test-rank-eval-with-mustache:integTestCluster#checkPrevious', task ':qa:smoke-test-rank-eval-with-mustache:integTestCluster#stopPrevious', task ':qa:smoke-test-rank-eval-with-mustache:integTestCluster#extract', task ':qa:smoke-test-rank-eval-with-mustache:integTestCluster#configure', task ':qa:smoke-test-rank-eval-with-mustache:integTestCluster#start', task ':qa:smoke-test-rank-eval-with-mustache:copyRestSpec', task ':qa:smoke-test-rank-eval-with-mustache:checkstyle', task ':qa:smoke-test-rank-eval-with-mustache:filepermissions', task ':qa:smoke-test-rank-eval-with-mustache:forbiddenPatterns', task ':qa:smoke-test-rank-eval-with-mustache:jarHell', task ':qa:smoke-test-rank-eval-with-mustache:licenseHeaders', task ':qa:smoke-test-rank-eval-with-mustache:loggerUsageCheck', task ':qa:smoke-test-rank-eval-with-mustache:namingConventions', task ':qa:smoke-test-rank-eval-with-mustache:thirdPartyAudit', task ':qa:smoke-test-rank-eval-with-mustache:precommit', task ':qa:smoke-test-rank-eval-with-mustache:integTestCluster#init', task ':qa:smoke-test-rank-eval-with-mustache:integTestCluster#wait', task ':qa:smoke-test-rank-eval-with-mustache:integTestRunner', task ':qa:smoke-test-rank-eval-with-mustache:integTestCluster#stop', task ':qa:smoke-test-rank-eval-with-mustache:integTest', task ':qa:smoke-test-rank-eval-with-mustache:check', task ':qa:smoke-test-rank-eval-with-mustache:build', task ':qa:smoke-test-reindex-with-all-modules:copyCheckstyleConf', task ':qa:smoke-test-reindex-with-all-modules:compileTestJava', task ':qa:smoke-test-reindex-with-all-modules:processTestResources', task ':qa:smoke-test-reindex-with-all-modules:testClasses', task ':qa:smoke-test-reindex-with-all-modules:checkstyleTest', task ':qa:smoke-test-reindex-with-all-modules:forbiddenApisTest', task ':qa:smoke-test-reindex-with-all-modules:forbiddenApis', task ':qa:smoke-test-reindex-with-all-modules:integTestCluster#prepareCluster.cleanShared', task ':qa:smoke-test-reindex-with-all-modules:integTestCluster#clean', task ':qa:smoke-test-reindex-with-all-modules:integTestCluster#createCwd', task ':qa:smoke-test-reindex-with-all-modules:integTestCluster#checkPrevious', task ':qa:smoke-test-reindex-with-all-modules:integTestCluster#stopPrevious', task ':qa:smoke-test-reindex-with-all-modules:integTestCluster#extract', task ':qa:smoke-test-reindex-with-all-modules:integTestCluster#configure', task ':qa:smoke-test-reindex-with-all-modules:integTestCluster#start', task ':qa:smoke-test-reindex-with-all-modules:copyRestSpec', task ':qa:smoke-test-reindex-with-all-modules:checkstyle', task ':qa:smoke-test-reindex-with-all-modules:filepermissions', task ':qa:smoke-test-reindex-with-all-modules:forbiddenPatterns', task ':qa:smoke-test-reindex-with-all-modules:jarHell', task ':qa:smoke-test-reindex-with-all-modules:licenseHeaders', task ':qa:smoke-test-reindex-with-all-modules:loggerUsageCheck', task ':qa:smoke-test-reindex-with-all-modules:namingConventions', task ':qa:smoke-test-reindex-with-all-modules:thirdPartyAudit', task ':qa:smoke-test-reindex-with-all-modules:precommit', task ':qa:smoke-test-reindex-with-all-modules:integTestCluster#init', task ':qa:smoke-test-reindex-with-all-modules:integTestCluster#wait', task ':qa:smoke-test-reindex-with-all-modules:integTestRunner', task ':qa:smoke-test-reindex-with-all-modules:integTestCluster#stop', task ':qa:smoke-test-reindex-with-all-modules:integTest', task ':qa:smoke-test-reindex-with-all-modules:check', task ':qa:smoke-test-reindex-with-all-modules:build', task ':qa:vagrant:compileJava', task ':qa:vagrant:processResources', task ':qa:vagrant:classes', task ':qa:vagrant:copyCheckstyleConf', task ':qa:vagrant:checkstyleMain', task ':qa:vagrant:compileTestJava', task ':qa:vagrant:processTestResources', task ':qa:vagrant:testClasses', task ':qa:vagrant:checkstyleTest', task ':qa:vagrant:forbiddenApisMain', task ':qa:vagrant:forbiddenApisTest', task ':qa:vagrant:forbiddenApis', task ':qa:vagrant:checkstyle', task ':qa:vagrant:dependencyLicenses', task ':qa:vagrant:filepermissions', task ':qa:vagrant:forbiddenPatterns', task ':qa:vagrant:jarHell', task ':qa:vagrant:licenseHeaders', task ':qa:vagrant:loggerUsageCheck', task ':qa:vagrant:namingConventions', task ':qa:vagrant:thirdPartyAudit', task ':qa:vagrant:precommit', task ':qa:vagrant:test', task ':qa:vagrant:jar', task ':qa:vagrant:extractJar', task ':qa:vagrant:verifyJarLicense', task ':qa:vagrant:verifyJarNotice', task ':qa:vagrant:javadoc', task ':qa:vagrant:javadocJar', task ':qa:vagrant:extractJavadocJar', task ':qa:vagrant:verifyJavadocJarLicense', task ':qa:vagrant:verifyJavadocJarNotice', task ':qa:vagrant:sourcesJar', task ':qa:vagrant:extractSourcesJar', task ':qa:vagrant:verifySourcesJarLicense', task ':qa:vagrant:verifySourcesJarNotice', task ':qa:vagrant:check', task ':qa:vagrant:build', task ':qa:verify-version-constants:copyCheckstyleConf', task ':qa:verify-version-constants:compileTestJava', task ':qa:verify-version-constants:processTestResources', task ':qa:verify-version-constants:testClasses', task ':qa:verify-version-constants:checkstyleTest', task ':qa:verify-version-constants:forbiddenApisTest', task ':qa:verify-version-constants:forbiddenApis', task ':qa:verify-version-constants:v6.4.0-SNAPSHOT#oldClusterTestCluster#prepareCluster.cleanShared', task ':qa:verify-version-constants:v6.4.0-SNAPSHOT#oldClusterTestCluster#clean', task ':qa:verify-version-constants:v6.4.0-SNAPSHOT#oldClusterTestCluster#createCwd', task ':qa:verify-version-constants:v6.4.0-SNAPSHOT#oldClusterTestCluster#checkPrevious', task ':qa:verify-version-constants:v6.4.0-SNAPSHOT#oldClusterTestCluster#stopPrevious', task ':qa:verify-version-constants:v6.4.0-SNAPSHOT#oldClusterTestCluster#extract', task ':qa:verify-version-constants:v6.4.0-SNAPSHOT#oldClusterTestCluster#configure', task ':qa:verify-version-constants:v6.4.0-SNAPSHOT#oldClusterTestCluster#start', task ':qa:verify-version-constants:copyRestSpec', task ':qa:verify-version-constants:checkstyle', task ':qa:verify-version-constants:filepermissions', task ':qa:verify-version-constants:forbiddenPatterns', task ':qa:verify-version-constants:jarHell', task ':qa:verify-version-constants:licenseHeaders', task ':qa:verify-version-constants:loggerUsageCheck', task ':qa:verify-version-constants:namingConventions', task ':qa:verify-version-constants:thirdPartyAudit', task ':qa:verify-version-constants:precommit', task ':qa:verify-version-constants:v6.4.0-SNAPSHOT#oldClusterTestCluster#init', task ':qa:verify-version-constants:v6.4.0-SNAPSHOT#oldClusterTestCluster#wait', task ':qa:verify-version-constants:v6.4.0-SNAPSHOT#oldClusterTestRunner', task ':qa:verify-version-constants:v6.4.0-SNAPSHOT#oldClusterTestCluster#stop', task ':qa:verify-version-constants:v6.4.0-SNAPSHOT#oldClusterTest', task ':qa:verify-version-constants:v6.4.0-SNAPSHOT#bwcTest', task ':qa:verify-version-constants:integTest', task ':qa:verify-version-constants:test', task ':qa:verify-version-constants:verifyDocsLuceneVersion', task ':qa:verify-version-constants:check', task ':qa:verify-version-constants:build', task ':qa:wildfly:compileJava', task ':qa:wildfly:processResources', task ':qa:wildfly:classes', task ':qa:wildfly:copyCheckstyleConf', task ':qa:wildfly:checkstyleMain', task ':qa:wildfly:compileTestJava', task ':qa:wildfly:processTestResources', task ':qa:wildfly:testClasses', task ':qa:wildfly:checkstyleTest', task ':qa:wildfly:forbiddenApisMain', task ':qa:wildfly:forbiddenApisTest', task ':qa:wildfly:forbiddenApis', task ':qa:wildfly:copyRestSpec', task ':qa:wildfly:checkstyle', task ':qa:wildfly:dependencyLicenses', task ':qa:wildfly:filepermissions', task ':qa:wildfly:forbiddenPatterns', task ':qa:wildfly:jarHell', task ':qa:wildfly:licenseHeaders', task ':qa:wildfly:loggerUsageCheck', task ':qa:wildfly:namingConventions', task ':qa:wildfly:thirdPartyAudit', task ':qa:wildfly:precommit', task ':qa:wildfly:integTestCluster#init', task ':qa:wildfly:integTestRunner', task ':qa:wildfly:integTest', task ':qa:wildfly:test', task ':qa:wildfly:jar', task ':qa:wildfly:extractJar', task ':qa:wildfly:verifyJarLicense', task ':qa:wildfly:verifyJarNotice', task ':qa:wildfly:javadoc', task ':qa:wildfly:javadocJar', task ':qa:wildfly:extractJavadocJar', task ':qa:wildfly:verifyJavadocJarLicense', task ':qa:wildfly:verifyJavadocJarNotice', task ':qa:wildfly:sourcesJar', task ':qa:wildfly:extractSourcesJar', task ':qa:wildfly:verifySourcesJarLicense', task ':qa:wildfly:verifySourcesJarNotice', task ':qa:wildfly:war', task ':qa:wildfly:extractWar', task ':qa:wildfly:verifyWarLicense', task ':qa:wildfly:verifyWarNotice', task ':qa:wildfly:check', task ':qa:wildfly:build', task ':server:cli:generatePomFileForNebulaPublication', task ':server:cli:javadocJar', task ':server:cli:sourcesJar', task ':server:cli:assemble', task ':server:cli:extractJar', task ':server:cli:verifyJarLicense', task ':server:cli:verifyJarNotice', task ':server:cli:extractJavadocJar', task ':server:cli:verifyJavadocJarLicense', task ':server:cli:verifyJavadocJarNotice', task ':server:cli:extractSourcesJar', task ':server:cli:verifySourcesJarLicense', task ':server:cli:verifySourcesJarNotice', task ':server:cli:check', task ':server:cli:build', task ':test:framework:generatePomFileForNebulaPublication', task ':test:framework:javadoc', task ':test:framework:javadocJar', task ':test:framework:sourcesJar', task ':test:framework:assemble', task ':test:framework:extractJar', task ':test:framework:verifyJarLicense', task ':test:framework:verifyJarNotice', task ':test:framework:extractJavadocJar', task ':test:framework:verifyJavadocJarLicense', task ':test:framework:verifyJavadocJarNotice', task ':test:framework:extractSourcesJar', task ':test:framework:verifySourcesJarLicense', task ':test:framework:verifySourcesJarNotice', task ':test:framework:check', task ':test:framework:build', task ':test:logger-usage:generatePomFileForNebulaPublication', task ':test:logger-usage:javadoc', task ':test:logger-usage:javadocJar', task ':test:logger-usage:sourcesJar', task ':test:logger-usage:assemble', task ':test:logger-usage:extractJar', task ':test:logger-usage:verifyJarLicense', task ':test:logger-usage:verifyJarNotice', task ':test:logger-usage:extractJavadocJar', task ':test:logger-usage:verifyJavadocJarLicense', task ':test:logger-usage:verifyJavadocJarNotice', task ':test:logger-usage:extractSourcesJar', task ':test:logger-usage:verifySourcesJarLicense', task ':test:logger-usage:verifySourcesJarNotice', task ':test:logger-usage:check', task ':test:logger-usage:build', task ':x-pack:docs:buildRestTests', task ':x-pack:docs:copyCheckstyleConf', task ':x-pack:plugin:core:compileTestJava', task ':x-pack:plugin:core:processTestResources', task ':x-pack:plugin:core:testClasses', task ':x-pack:plugin:core:testJar', task ':x-pack:plugin:compileTestJava', task ':x-pack:plugin:createNodeKeyStore', task ':x-pack:plugin:processTestResources', task ':x-pack:plugin:testClasses', task ':x-pack:plugin:testJar', task ':x-pack:docs:compileTestJava', task ':x-pack:docs:processTestResources', task ':x-pack:docs:testClasses', task ':x-pack:docs:checkstyleTest', task ':x-pack:docs:forbiddenApisTest', task ':x-pack:docs:forbiddenApis', task ':x-pack:docs:integTestCluster#prepareCluster.cleanShared', task ':x-pack:docs:integTestCluster#clean', task ':x-pack:docs:integTestCluster#createCwd', task ':x-pack:docs:integTestCluster#checkPrevious', task ':x-pack:docs:integTestCluster#stopPrevious', task ':x-pack:docs:integTestCluster#extract', task ':x-pack:docs:integTestCluster#configure', task ':x-pack:docs:integTestCluster#setupTestAdmin', task ':x-pack:docs:integTestCluster#start', task ':x-pack:docs:copyRestSpec', task ':x-pack:docs:checkstyle', task ':x-pack:docs:filepermissions', task ':x-pack:docs:forbiddenPatterns', task ':x-pack:docs:jarHell', task ':x-pack:docs:licenseHeaders', task ':x-pack:docs:loggerUsageCheck', task ':x-pack:docs:namingConventions', task ':x-pack:docs:thirdPartyAudit', task ':x-pack:docs:precommit', task ':x-pack:plugin:integTestCluster#prepareCluster.cleanShared', task ':x-pack:plugin:integTestCluster#clean', task ':x-pack:plugin:integTestCluster#createCwd', task ':x-pack:plugin:integTestCluster#checkPrevious', task ':x-pack:plugin:integTestCluster#stopPrevious', task ':x-pack:plugin:integTestCluster#extract', task ':x-pack:plugin:integTestCluster#configure', task ':x-pack:plugin:integTestCluster#createKeystore', task ':x-pack:plugin:integTestCluster#addToKeystore#xpack.security.transport.ssl.keystore.secure_password', task ':x-pack:plugin:integTestCluster#addToKeystore#bootstrap.password', task ':x-pack:plugin:integTestCluster#extraConfig', task ':x-pack:plugin:integTestCluster#setupTestUser', task ':x-pack:plugin:integTestCluster#start', task ':x-pack:plugin:copyRestSpec', task ':x-pack:plugin:copyCheckstyleConf', task ':x-pack:plugin:checkstyleTest', task ':x-pack:plugin:checkstyle', task ':x-pack:plugin:filepermissions', task ':x-pack:plugin:forbiddenApisTest', task ':x-pack:plugin:forbiddenApis', task ':x-pack:plugin:forbiddenPatterns', task ':x-pack:plugin:jarHell', task ':x-pack:plugin:licenseHeaders', task ':x-pack:plugin:loggerUsageCheck', task ':x-pack:plugin:namingConventions', task ':x-pack:plugin:thirdPartyAudit', task ':x-pack:plugin:precommit', task ':x-pack:plugin:core:copyRestSpec', task ':x-pack:plugin:core:copyCheckstyleConf', task ':x-pack:plugin:core:checkstyleMain', task ':x-pack:plugin:core:checkstyleTest', task ':x-pack:plugin:core:checkstyle', task ':x-pack:plugin:core:dependencyLicenses', task ':x-pack:plugin:core:filepermissions', task ':x-pack:plugin:core:forbiddenApisMain', task ':x-pack:plugin:core:forbiddenApisTest', task ':x-pack:plugin:core:forbiddenApis', task ':x-pack:plugin:core:forbiddenPatterns', task ':x-pack:plugin:core:jarHell', task ':x-pack:plugin:core:licenseHeaders', task ':x-pack:plugin:core:loggerUsageCheck', task ':x-pack:plugin:core:namingConventions', task ':x-pack:plugin:core:thirdPartyAudit', task ':x-pack:plugin:core:precommit', task ':x-pack:plugin:core:test', task ':x-pack:plugin:core:integTestCluster#init', task ':x-pack:plugin:core:integTestRunner', task ':x-pack:plugin:core:integTest', task ':x-pack:plugin:integTestCluster#init', task ':x-pack:plugin:integTestCluster#wait', task ':x-pack:plugin:integTestRunner', task ':x-pack:plugin:integTestCluster#stop', task ':x-pack:plugin:integTest', task ':x-pack:docs:integTestCluster#init', task ':x-pack:docs:integTestCluster#wait', task ':x-pack:docs:integTestRunner', task ':x-pack:docs:integTestCluster#stop', task ':x-pack:docs:integTest', task ':x-pack:docs:check', task ':x-pack:docs:build', task ':x-pack:license-tools:compileJava', task ':x-pack:license-tools:processResources', task ':x-pack:license-tools:classes', task ':x-pack:license-tools:jar', task ':x-pack:license-tools:buildZip', task ':x-pack:plugin:core:javadoc', task ':x-pack:license-tools:javadoc', task ':x-pack:license-tools:javadocJar', task ':x-pack:license-tools:sourcesJar', task ':x-pack:license-tools:assemble', task ':x-pack:license-tools:copyCheckstyleConf', task ':x-pack:license-tools:checkstyleMain', task ':x-pack:license-tools:compileTestJava', task ':x-pack:license-tools:processTestResources', task ':x-pack:license-tools:testClasses', task ':x-pack:license-tools:checkstyleTest', task ':x-pack:license-tools:forbiddenApisMain', task ':x-pack:license-tools:forbiddenApisTest', task ':x-pack:license-tools:forbiddenApis', task ':x-pack:license-tools:checkstyle', task ':x-pack:license-tools:dependencyLicenses', task ':x-pack:license-tools:filepermissions', task ':x-pack:license-tools:forbiddenPatterns', task ':x-pack:license-tools:jarHell', task ':x-pack:license-tools:licenseHeaders', task ':x-pack:license-tools:loggerUsageCheck', task ':x-pack:license-tools:namingConventions', task ':x-pack:license-tools:thirdPartyAudit', task ':x-pack:license-tools:precommit', task ':x-pack:license-tools:test', task ':x-pack:license-tools:extractJar', task ':x-pack:license-tools:verifyJarLicense', task ':x-pack:license-tools:verifyJarNotice', task ':x-pack:license-tools:extractJavadocJar', task ':x-pack:license-tools:verifyJavadocJarLicense', task ':x-pack:license-tools:verifyJavadocJarNotice', task ':x-pack:license-tools:extractSourcesJar', task ':x-pack:license-tools:verifySourcesJarLicense', task ':x-pack:license-tools:verifySourcesJarNotice', task ':x-pack:license-tools:check', task ':x-pack:license-tools:build', task ':x-pack:plugin:generatePomFileForZipPublication', task ':x-pack:plugin:generatePomFileForZipRealPublication', task ':x-pack:plugin:assemble', task ':x-pack:plugin:extractTestJar', task ':x-pack:plugin:verifyTestJarLicense', task ':x-pack:plugin:verifyTestJarNotice', task ':x-pack:plugin:check', task ':x-pack:plugin:build', task ':x-pack:transport-client:generatePomFileForNebulaPublication', task ':x-pack:transport-client:compileJava', task ':x-pack:transport-client:processResources', task ':x-pack:transport-client:classes', task ':x-pack:transport-client:jar', task ':x-pack:transport-client:javadoc', task ':x-pack:transport-client:javadocJar', task ':x-pack:transport-client:sourcesJar', task ':x-pack:transport-client:assemble', task ':x-pack:transport-client:copyCheckstyleConf', task ':x-pack:transport-client:checkstyleMain', task ':x-pack:transport-client:compileTestJava', task ':x-pack:transport-client:processTestResources', task ':x-pack:transport-client:testClasses', task ':x-pack:transport-client:checkstyleTest', task ':x-pack:transport-client:forbiddenApisMain', task ':x-pack:transport-client:forbiddenApisTest', task ':x-pack:transport-client:forbiddenApis', task ':x-pack:transport-client:checkstyle', task ':x-pack:transport-client:dependencyLicenses', task ':x-pack:transport-client:filepermissions', task ':x-pack:transport-client:forbiddenPatterns', task ':x-pack:transport-client:jarHell', task ':x-pack:transport-client:licenseHeaders', task ':x-pack:transport-client:loggerUsageCheck', task ':x-pack:transport-client:namingConventions', task ':x-pack:transport-client:thirdPartyAudit', task ':x-pack:transport-client:precommit', task ':x-pack:transport-client:test', task ':x-pack:transport-client:extractJar', task ':x-pack:transport-client:verifyJarLicense', task ':x-pack:transport-client:verifyJarNotice', task ':x-pack:transport-client:extractJavadocJar', task ':x-pack:transport-client:verifyJavadocJarLicense', task ':x-pack:transport-client:verifyJavadocJarNotice', task ':x-pack:transport-client:extractSourcesJar', task ':x-pack:transport-client:verifySourcesJarLicense', task ':x-pack:transport-client:verifySourcesJarNotice', task ':x-pack:transport-client:check', task ':x-pack:transport-client:build', task ':distribution:archives:integ-test-zip:checkExtraction', task ':distribution:archives:integ-test-zip:checkLicense', task ':distribution:archives:integ-test-zip:checkNotice', task ':distribution:archives:integ-test-zip:check', task ':distribution:archives:integ-test-zip:build', task ':distribution:archives:buildOssTar', task ':distribution:archives:oss-tar:distTar', task ':distribution:archives:oss-tar:distZip', task ':distribution:archives:oss-tar:assemble', task ':distribution:archives:oss-tar:checkExtraction', task ':distribution:archives:oss-tar:checkLicense', task ':distribution:archives:oss-tar:checkNotice', task ':distribution:archives:oss-tar:check', task ':distribution:archives:oss-tar:build', task ':distribution:archives:oss-zip:distTar', task ':distribution:archives:oss-zip:distZip', task ':distribution:archives:oss-zip:generatePomFileForNebulaPublication', task ':distribution:archives:oss-zip:generatePomFileForNebulaRealPomPublication', task ':distribution:archives:oss-zip:assemble', task ':distribution:archives:oss-zip:checkExtraction', task ':distribution:archives:oss-zip:checkLicense', task ':distribution:archives:oss-zip:checkNotice', task ':distribution:archives:oss-zip:check', task ':distribution:archives:oss-zip:build', task ':distribution:archives:buildTar', task ':distribution:archives:tar:distTar', task ':distribution:archives:tar:distZip', task ':distribution:archives:tar:assemble', task ':distribution:archives:tar:checkExtraction', task ':distribution:archives:tar:checkLicense', task ':distribution:archives:tar:checkMlCppNotice', task ':distribution:archives:tar:checkNotice', task ':distribution:archives:tar:check', task ':distribution:archives:tar:build', task ':distribution:archives:zip:distTar', task ':distribution:archives:zip:distZip', task ':distribution:archives:zip:generatePomFileForNebulaPublication', task ':distribution:archives:zip:generatePomFileForNebulaRealPomPublication', task ':distribution:archives:zip:assemble', task ':distribution:archives:zip:checkExtraction', task ':distribution:archives:zip:checkLicense', task ':distribution:archives:zip:checkMlCppNotice', task ':distribution:archives:zip:checkNotice', task ':distribution:archives:zip:check', task ':distribution:archives:zip:build', task ':distribution:bwc:maintenance-bugfix-snapshot:check', task ':distribution:bwc:maintenance-bugfix-snapshot:build', task ':distribution:bwc:next-bugfix-snapshot:check', task ':distribution:bwc:next-bugfix-snapshot:build', task ':distribution:bwc:next-minor-snapshot:check', task ':distribution:bwc:next-minor-snapshot:build', task ':distribution:bwc:staged-minor-snapshot:check', task ':distribution:bwc:staged-minor-snapshot:build', task ':distribution:packages:processDebFiles', task ':distribution:packages:buildDeb', task ':distribution:packages:deb:distTar', task ':distribution:packages:deb:distZip', task ':distribution:packages:deb:assemble', task ':distribution:packages:deb:checkExtraction', task ':distribution:packages:deb:checkLicense', task ':distribution:packages:deb:checkLicenseMetadata', task ':distribution:packages:deb:checkNotice', task ':distribution:packages:deb:check', task ':distribution:packages:deb:build', task ':distribution:packages:processOssDebFiles', task ':distribution:packages:buildOssDeb', task ':distribution:packages:oss-deb:distTar', task ':distribution:packages:oss-deb:distZip', task ':distribution:packages:oss-deb:assemble', task ':distribution:packages:oss-deb:checkExtraction', task ':distribution:packages:oss-deb:checkLicense', task ':distribution:packages:oss-deb:checkLicenseMetadata', task ':distribution:packages:oss-deb:checkNotice', task ':distribution:packages:oss-deb:check', task ':distribution:packages:oss-deb:build', task ':distribution:packages:processOssRpmFiles', task ':distribution:packages:buildOssRpm', task ':distribution:packages:oss-rpm:distTar', task ':distribution:packages:oss-rpm:distZip', task ':distribution:packages:oss-rpm:assemble', task ':distribution:packages:oss-rpm:checkExtraction', task ':distribution:packages:oss-rpm:checkLicense', task ':distribution:packages:oss-rpm:checkLicenseMetadata', task ':distribution:packages:oss-rpm:checkNotice', task ':distribution:packages:oss-rpm:check', task ':distribution:packages:oss-rpm:build', task ':distribution:packages:processRpmFiles', task ':distribution:packages:buildRpm', task ':distribution:packages:rpm:distTar', task ':distribution:packages:rpm:distZip', task ':distribution:packages:rpm:assemble', task ':distribution:packages:rpm:checkExtraction', task ':distribution:packages:rpm:checkLicense', task ':distribution:packages:rpm:checkLicenseMetadata', task ':distribution:packages:rpm:checkNotice', task ':distribution:packages:rpm:check', task ':distribution:packages:rpm:build', task ':distribution:tools:launchers:javadoc', task ':distribution:tools:launchers:javadocJar', task ':distribution:tools:launchers:sourcesJar', task ':distribution:tools:launchers:assemble', task ':distribution:tools:launchers:animalsnifferMain', task ':distribution:tools:launchers:compileTestJava', task ':distribution:tools:launchers:processTestResources', task ':distribution:tools:launchers:testClasses', task ':distribution:tools:launchers:animalsnifferTest', task ':distribution:tools:launchers:copyCheckstyleConf', task ':distribution:tools:launchers:checkstyleMain', task ':distribution:tools:launchers:checkstyleTest', task ':distribution:tools:launchers:forbiddenApisMain', task ':distribution:tools:launchers:forbiddenApisTest', task ':distribution:tools:launchers:forbiddenApis', task ':distribution:tools:launchers:checkstyle', task ':distribution:tools:launchers:dependencyLicenses', task ':distribution:tools:launchers:filepermissions', task ':distribution:tools:launchers:forbiddenPatterns', task ':distribution:tools:launchers:jarHell', task ':distribution:tools:launchers:licenseHeaders', task ':distribution:tools:launchers:loggerUsageCheck', task ':distribution:tools:launchers:namingConventions', task ':distribution:tools:launchers:thirdPartyAudit', task ':distribution:tools:launchers:precommit', task ':distribution:tools:launchers:test', task ':distribution:tools:launchers:extractJar', task ':distribution:tools:launchers:verifyJarLicense', task ':distribution:tools:launchers:verifyJarNotice', task ':distribution:tools:launchers:extractJavadocJar', task ':distribution:tools:launchers:verifyJavadocJarLicense', task ':distribution:tools:launchers:verifyJavadocJarNotice', task ':distribution:tools:launchers:extractSourcesJar', task ':distribution:tools:launchers:verifySourcesJarLicense', task ':distribution:tools:launchers:verifySourcesJarNotice', task ':distribution:tools:launchers:check', task ':distribution:tools:launchers:build', task ':distribution:tools:plugin-cli:javadoc', task ':distribution:tools:plugin-cli:javadocJar', task ':distribution:tools:plugin-cli:sourcesJar', task ':distribution:tools:plugin-cli:assemble', task ':distribution:tools:plugin-cli:copyCheckstyleConf', task ':distribution:tools:plugin-cli:checkstyleMain', task ':distribution:tools:plugin-cli:compileTestJava', task ':distribution:tools:plugin-cli:processTestResources', task ':distribution:tools:plugin-cli:testClasses', task ':distribution:tools:plugin-cli:checkstyleTest', task ':distribution:tools:plugin-cli:forbiddenApisMain', task ':distribution:tools:plugin-cli:forbiddenApisTest', task ':distribution:tools:plugin-cli:forbiddenApis', task ':distribution:tools:plugin-cli:checkstyle', task ':distribution:tools:plugin-cli:dependencyLicenses', task ':distribution:tools:plugin-cli:filepermissions', task ':distribution:tools:plugin-cli:forbiddenPatterns', task ':distribution:tools:plugin-cli:jarHell', task ':distribution:tools:plugin-cli:licenseHeaders', task ':distribution:tools:plugin-cli:loggerUsageCheck', task ':distribution:tools:plugin-cli:namingConventions', task ':distribution:tools:plugin-cli:thirdPartyAudit', task ':distribution:tools:plugin-cli:precommit', task ':distribution:tools:plugin-cli:test', task ':distribution:tools:plugin-cli:extractJar', task ':distribution:tools:plugin-cli:verifyJarLicense', task ':distribution:tools:plugin-cli:verifyJarNotice', task ':distribution:tools:plugin-cli:extractJavadocJar', task ':distribution:tools:plugin-cli:verifyJavadocJarLicense', task ':distribution:tools:plugin-cli:verifyJavadocJarNotice', task ':distribution:tools:plugin-cli:extractSourcesJar', task ':distribution:tools:plugin-cli:verifySourcesJarLicense', task ':distribution:tools:plugin-cli:verifySourcesJarNotice', task ':distribution:tools:plugin-cli:check', task ':distribution:tools:plugin-cli:build', task ':example-plugins:meta-plugin:dummy-plugin1:copyCheckstyleConf', task ':example-plugins:meta-plugin:dummy-plugin1:checkstyleMain', task ':example-plugins:meta-plugin:dummy-plugin1:compileTestJava', task ':example-plugins:meta-plugin:dummy-plugin1:processTestResources', task ':example-plugins:meta-plugin:dummy-plugin1:testClasses', task ':example-plugins:meta-plugin:dummy-plugin1:checkstyleTest', task ':example-plugins:meta-plugin:dummy-plugin1:forbiddenApisMain', task ':example-plugins:meta-plugin:dummy-plugin1:forbiddenApisTest', task ':example-plugins:meta-plugin:dummy-plugin1:forbiddenApis', task ':example-plugins:meta-plugin:dummy-plugin1:integTestCluster#prepareCluster.cleanShared', task ':example-plugins:meta-plugin:dummy-plugin1:integTestCluster#clean', task ':example-plugins:meta-plugin:dummy-plugin1:integTestCluster#createCwd', task ':example-plugins:meta-plugin:dummy-plugin1:integTestCluster#checkPrevious', task ':example-plugins:meta-plugin:dummy-plugin1:integTestCluster#stopPrevious', task ':example-plugins:meta-plugin:dummy-plugin1:integTestCluster#extract', task ':example-plugins:meta-plugin:dummy-plugin1:integTestCluster#configure', task ':example-plugins:meta-plugin:dummy-plugin1:integTestCluster#copyPlugins', task ':example-plugins:meta-plugin:dummy-plugin1:integTestCluster#installDummyPlugin1Plugin', task ':example-plugins:meta-plugin:dummy-plugin1:integTestCluster#start', task ':example-plugins:meta-plugin:dummy-plugin1:copyRestSpec', task ':example-plugins:meta-plugin:dummy-plugin1:checkstyle', task ':example-plugins:meta-plugin:dummy-plugin1:dependencyLicenses', task ':example-plugins:meta-plugin:dummy-plugin1:filepermissions', task ':example-plugins:meta-plugin:dummy-plugin1:forbiddenPatterns', task ':example-plugins:meta-plugin:dummy-plugin1:jarHell', task ':example-plugins:meta-plugin:dummy-plugin1:licenseHeaders', task ':example-plugins:meta-plugin:dummy-plugin1:loggerUsageCheck', task ':example-plugins:meta-plugin:dummy-plugin1:namingConventions', task ':example-plugins:meta-plugin:dummy-plugin1:thirdPartyAudit', task ':example-plugins:meta-plugin:dummy-plugin1:precommit', task ':example-plugins:meta-plugin:dummy-plugin1:test', task ':example-plugins:meta-plugin:dummy-plugin1:integTestCluster#init', task ':example-plugins:meta-plugin:dummy-plugin1:integTestCluster#wait', task ':example-plugins:meta-plugin:dummy-plugin1:integTestRunner', task ':example-plugins:meta-plugin:dummy-plugin1:integTestCluster#stop', task ':example-plugins:meta-plugin:dummy-plugin1:integTest', task ':example-plugins:meta-plugin:dummy-plugin1:extractJar', task ':example-plugins:meta-plugin:dummy-plugin1:verifyJarLicense', task ':example-plugins:meta-plugin:dummy-plugin1:verifyJarNotice', task ':example-plugins:meta-plugin:dummy-plugin1:javadoc', task ':example-plugins:meta-plugin:dummy-plugin1:javadocJar', task ':example-plugins:meta-plugin:dummy-plugin1:extractJavadocJar', task ':example-plugins:meta-plugin:dummy-plugin1:verifyJavadocJarLicense', task ':example-plugins:meta-plugin:dummy-plugin1:verifyJavadocJarNotice', task ':example-plugins:meta-plugin:dummy-plugin1:sourcesJar', task ':example-plugins:meta-plugin:dummy-plugin1:extractSourcesJar', task ':example-plugins:meta-plugin:dummy-plugin1:verifySourcesJarLicense', task ':example-plugins:meta-plugin:dummy-plugin1:verifySourcesJarNotice', task ':example-plugins:meta-plugin:dummy-plugin1:check', task ':example-plugins:meta-plugin:dummy-plugin1:build', task ':example-plugins:meta-plugin:dummy-plugin2:copyCheckstyleConf', task ':example-plugins:meta-plugin:dummy-plugin2:checkstyleMain', task ':example-plugins:meta-plugin:dummy-plugin2:compileTestJava', task ':example-plugins:meta-plugin:dummy-plugin2:processTestResources', task ':example-plugins:meta-plugin:dummy-plugin2:testClasses', task ':example-plugins:meta-plugin:dummy-plugin2:checkstyleTest', task ':example-plugins:meta-plugin:dummy-plugin2:forbiddenApisMain', task ':example-plugins:meta-plugin:dummy-plugin2:forbiddenApisTest', task ':example-plugins:meta-plugin:dummy-plugin2:forbiddenApis', task ':example-plugins:meta-plugin:dummy-plugin2:integTestCluster#prepareCluster.cleanShared', task ':example-plugins:meta-plugin:dummy-plugin2:integTestCluster#clean', task ':example-plugins:meta-plugin:dummy-plugin2:integTestCluster#createCwd', task ':example-plugins:meta-plugin:dummy-plugin2:integTestCluster#checkPrevious', task ':example-plugins:meta-plugin:dummy-plugin2:integTestCluster#stopPrevious', task ':example-plugins:meta-plugin:dummy-plugin2:integTestCluster#extract', task ':example-plugins:meta-plugin:dummy-plugin2:integTestCluster#configure', task ':example-plugins:meta-plugin:dummy-plugin2:integTestCluster#copyPlugins', task ':example-plugins:meta-plugin:dummy-plugin2:integTestCluster#installDummyPlugin2Plugin', task ':example-plugins:meta-plugin:dummy-plugin2:integTestCluster#start', task ':example-plugins:meta-plugin:dummy-plugin2:copyRestSpec', task ':example-plugins:meta-plugin:dummy-plugin2:checkstyle', task ':example-plugins:meta-plugin:dummy-plugin2:dependencyLicenses', task ':example-plugins:meta-plugin:dummy-plugin2:filepermissions', task ':example-plugins:meta-plugin:dummy-plugin2:forbiddenPatterns', task ':example-plugins:meta-plugin:dummy-plugin2:jarHell', task ':example-plugins:meta-plugin:dummy-plugin2:licenseHeaders', task ':example-plugins:meta-plugin:dummy-plugin2:loggerUsageCheck', task ':example-plugins:meta-plugin:dummy-plugin2:namingConventions', task ':example-plugins:meta-plugin:dummy-plugin2:thirdPartyAudit', task ':example-plugins:meta-plugin:dummy-plugin2:precommit', task ':example-plugins:meta-plugin:dummy-plugin2:test', task ':example-plugins:meta-plugin:dummy-plugin2:integTestCluster#init', task ':example-plugins:meta-plugin:dummy-plugin2:integTestCluster#wait', task ':example-plugins:meta-plugin:dummy-plugin2:integTestRunner', task ':example-plugins:meta-plugin:dummy-plugin2:integTestCluster#stop', task ':example-plugins:meta-plugin:dummy-plugin2:integTest', task ':example-plugins:meta-plugin:dummy-plugin2:extractJar', task ':example-plugins:meta-plugin:dummy-plugin2:verifyJarLicense', task ':example-plugins:meta-plugin:dummy-plugin2:verifyJarNotice', task ':example-plugins:meta-plugin:dummy-plugin2:javadoc', task ':example-plugins:meta-plugin:dummy-plugin2:javadocJar', task ':example-plugins:meta-plugin:dummy-plugin2:extractJavadocJar', task ':example-plugins:meta-plugin:dummy-plugin2:verifyJavadocJarLicense', task ':example-plugins:meta-plugin:dummy-plugin2:verifyJavadocJarNotice', task ':example-plugins:meta-plugin:dummy-plugin2:sourcesJar', task ':example-plugins:meta-plugin:dummy-plugin2:extractSourcesJar', task ':example-plugins:meta-plugin:dummy-plugin2:verifySourcesJarLicense', task ':example-plugins:meta-plugin:dummy-plugin2:verifySourcesJarNotice', task ':example-plugins:meta-plugin:dummy-plugin2:check', task ':example-plugins:meta-plugin:dummy-plugin2:build', task ':modules:lang-painless:spi:generatePomFileForNebulaPublication', task ':modules:lang-painless:spi:javadocJar', task ':modules:lang-painless:spi:sourcesJar', task ':modules:lang-painless:spi:assemble', task ':modules:lang-painless:spi:extractJar', task ':modules:lang-painless:spi:verifyJarLicense', task ':modules:lang-painless:spi:verifyJarNotice', task ':modules:lang-painless:spi:extractJavadocJar', task ':modules:lang-painless:spi:verifyJavadocJarLicense', task ':modules:lang-painless:spi:verifyJavadocJarNotice', task ':modules:lang-painless:spi:extractSourcesJar', task ':modules:lang-painless:spi:verifySourcesJarLicense', task ':modules:lang-painless:spi:verifySourcesJarNotice', task ':modules:lang-painless:spi:check', task ':modules:lang-painless:spi:build', task ':test:fixtures:example-fixture:extractJar', task ':test:fixtures:example-fixture:verifyJarLicense', task ':test:fixtures:example-fixture:verifyJarNotice', task ':test:fixtures:example-fixture:javadoc', task ':test:fixtures:example-fixture:javadocJar', task ':test:fixtures:example-fixture:extractJavadocJar', task ':test:fixtures:example-fixture:verifyJavadocJarLicense', task ':test:fixtures:example-fixture:verifyJavadocJarNotice', task ':test:fixtures:example-fixture:sourcesJar', task ':test:fixtures:example-fixture:extractSourcesJar', task ':test:fixtures:example-fixture:verifySourcesJarLicense', task ':test:fixtures:example-fixture:verifySourcesJarNotice', task ':test:fixtures:example-fixture:check', task ':test:fixtures:example-fixture:build', task ':test:fixtures:hdfs-fixture:generatePomFileForNebulaPublication', task ':test:fixtures:hdfs-fixture:jar', task ':test:fixtures:hdfs-fixture:javadoc', task ':test:fixtures:hdfs-fixture:javadocJar', task ':test:fixtures:hdfs-fixture:sourcesJar', task ':test:fixtures:hdfs-fixture:assemble', task ':test:fixtures:hdfs-fixture:extractJar', task ':test:fixtures:hdfs-fixture:verifyJarLicense', task ':test:fixtures:hdfs-fixture:verifyJarNotice', task ':test:fixtures:hdfs-fixture:extractJavadocJar', task ':test:fixtures:hdfs-fixture:verifyJavadocJarLicense', task ':test:fixtures:hdfs-fixture:verifyJavadocJarNotice', task ':test:fixtures:hdfs-fixture:extractSourcesJar', task ':test:fixtures:hdfs-fixture:verifySourcesJarLicense', task ':test:fixtures:hdfs-fixture:verifySourcesJarNotice', task ':test:fixtures:hdfs-fixture:check', task ':test:fixtures:hdfs-fixture:build', task ':test:fixtures:krb5kdc-fixture:generatePomFileForNebulaPublication', task ':test:fixtures:krb5kdc-fixture:compileJava', task ':test:fixtures:krb5kdc-fixture:processResources', task ':test:fixtures:krb5kdc-fixture:classes', task ':test:fixtures:krb5kdc-fixture:jar', task ':test:fixtures:krb5kdc-fixture:javadoc', task ':test:fixtures:krb5kdc-fixture:javadocJar', task ':test:fixtures:krb5kdc-fixture:sourcesJar', task ':test:fixtures:krb5kdc-fixture:assemble', task ':test:fixtures:krb5kdc-fixture:copyCheckstyleConf', task ':test:fixtures:krb5kdc-fixture:checkstyleMain', task ':test:fixtures:krb5kdc-fixture:compileTestJava', task ':test:fixtures:krb5kdc-fixture:processTestResources', task ':test:fixtures:krb5kdc-fixture:testClasses', task ':test:fixtures:krb5kdc-fixture:checkstyleTest', task ':test:fixtures:krb5kdc-fixture:forbiddenApisMain', task ':test:fixtures:krb5kdc-fixture:forbiddenApisTest', task ':test:fixtures:krb5kdc-fixture:forbiddenApis', task ':test:fixtures:krb5kdc-fixture:checkstyle', task ':test:fixtures:krb5kdc-fixture:dependencyLicenses', task ':test:fixtures:krb5kdc-fixture:filepermissions', task ':test:fixtures:krb5kdc-fixture:forbiddenPatterns', task ':test:fixtures:krb5kdc-fixture:jarHell', task ':test:fixtures:krb5kdc-fixture:licenseHeaders', task ':test:fixtures:krb5kdc-fixture:loggerUsageCheck', task ':test:fixtures:krb5kdc-fixture:namingConventions', task ':test:fixtures:krb5kdc-fixture:thirdPartyAudit', task ':test:fixtures:krb5kdc-fixture:precommit', task ':test:fixtures:krb5kdc-fixture:test', task ':test:fixtures:krb5kdc-fixture:extractJar', task ':test:fixtures:krb5kdc-fixture:verifyJarLicense', task ':test:fixtures:krb5kdc-fixture:verifyJarNotice', task ':test:fixtures:krb5kdc-fixture:extractJavadocJar', task ':test:fixtures:krb5kdc-fixture:verifyJavadocJarLicense', task ':test:fixtures:krb5kdc-fixture:verifyJavadocJarNotice', task ':test:fixtures:krb5kdc-fixture:extractSourcesJar', task ':test:fixtures:krb5kdc-fixture:verifySourcesJarLicense', task ':test:fixtures:krb5kdc-fixture:verifySourcesJarNotice', task ':test:fixtures:krb5kdc-fixture:check', task ':test:fixtures:krb5kdc-fixture:build', task ':test:fixtures:old-elasticsearch:generatePomFileForNebulaPublication', task ':test:fixtures:old-elasticsearch:jar', task ':test:fixtures:old-elasticsearch:javadoc', task ':test:fixtures:old-elasticsearch:javadocJar', task ':test:fixtures:old-elasticsearch:sourcesJar', task ':test:fixtures:old-elasticsearch:assemble', task ':test:fixtures:old-elasticsearch:extractJar', task ':test:fixtures:old-elasticsearch:verifyJarLicense', task ':test:fixtures:old-elasticsearch:verifyJarNotice', task ':test:fixtures:old-elasticsearch:extractJavadocJar', task ':test:fixtures:old-elasticsearch:verifyJavadocJarLicense', task ':test:fixtures:old-elasticsearch:verifyJavadocJarNotice', task ':test:fixtures:old-elasticsearch:extractSourcesJar', task ':test:fixtures:old-elasticsearch:verifySourcesJarLicense', task ':test:fixtures:old-elasticsearch:verifySourcesJarNotice', task ':test:fixtures:old-elasticsearch:check', task ':test:fixtures:old-elasticsearch:build', task ':x-pack:plugin:core:generatePomFileForZipPublication', task ':x-pack:plugin:core:generatePomFileForZipRealPublication', task ':x-pack:plugin:core:javadocJar', task ':x-pack:plugin:core:sourcesJar', task ':x-pack:plugin:core:assemble', task ':x-pack:plugin:core:extractJar', task ':x-pack:plugin:core:verifyJarLicense', task ':x-pack:plugin:core:verifyJarNotice', task ':x-pack:plugin:core:extractJavadocJar', task ':x-pack:plugin:core:verifyJavadocJarLicense', task ':x-pack:plugin:core:verifyJavadocJarNotice', task ':x-pack:plugin:core:extractSourcesJar', task ':x-pack:plugin:core:verifySourcesJarLicense', task ':x-pack:plugin:core:verifySourcesJarNotice', task ':x-pack:plugin:core:extractTestJar', task ':x-pack:plugin:core:verifyTestJarLicense', task ':x-pack:plugin:core:verifyTestJarNotice', task ':x-pack:plugin:core:check', task ':x-pack:plugin:core:build', task ':x-pack:plugin:deprecation:generatePomFileForZipPublication', task ':x-pack:plugin:deprecation:generatePomFileForZipRealPublication', task ':x-pack:plugin:deprecation:javadoc', task ':x-pack:plugin:deprecation:javadocJar', task ':x-pack:plugin:deprecation:sourcesJar', task ':x-pack:plugin:deprecation:assemble', task ':x-pack:plugin:deprecation:copyCheckstyleConf', task ':x-pack:plugin:deprecation:checkstyleMain', task ':x-pack:plugin:deprecation:compileTestJava', task ':x-pack:plugin:deprecation:processTestResources', task ':x-pack:plugin:deprecation:testClasses', task ':x-pack:plugin:deprecation:checkstyleTest', task ':x-pack:plugin:deprecation:forbiddenApisMain', task ':x-pack:plugin:deprecation:forbiddenApisTest', task ':x-pack:plugin:deprecation:forbiddenApis', task ':x-pack:plugin:deprecation:copyRestSpec', task ':x-pack:plugin:deprecation:checkstyle', task ':x-pack:plugin:deprecation:dependencyLicenses', task ':x-pack:plugin:deprecation:filepermissions', task ':x-pack:plugin:deprecation:forbiddenPatterns', task ':x-pack:plugin:deprecation:jarHell', task ':x-pack:plugin:deprecation:licenseHeaders', task ':x-pack:plugin:deprecation:loggerUsageCheck', task ':x-pack:plugin:deprecation:namingConventions', task ':x-pack:plugin:deprecation:thirdPartyAudit', task ':x-pack:plugin:deprecation:precommit', task ':x-pack:plugin:deprecation:test', task ':x-pack:plugin:deprecation:integTestCluster#init', task ':x-pack:plugin:deprecation:integTestRunner', task ':x-pack:plugin:deprecation:integTest', task ':x-pack:plugin:deprecation:extractJar', task ':x-pack:plugin:deprecation:verifyJarLicense', task ':x-pack:plugin:deprecation:verifyJarNotice', task ':x-pack:plugin:deprecation:extractJavadocJar', task ':x-pack:plugin:deprecation:verifyJavadocJarLicense', task ':x-pack:plugin:deprecation:verifyJavadocJarNotice', task ':x-pack:plugin:deprecation:extractSourcesJar', task ':x-pack:plugin:deprecation:verifySourcesJarLicense', task ':x-pack:plugin:deprecation:verifySourcesJarNotice', task ':x-pack:plugin:deprecation:check', task ':x-pack:plugin:deprecation:build', task ':x-pack:plugin:graph:generatePomFileForZipPublication', task ':x-pack:plugin:graph:generatePomFileForZipRealPublication', task ':x-pack:plugin:graph:javadoc', task ':x-pack:plugin:graph:javadocJar', task ':x-pack:plugin:graph:sourcesJar', task ':x-pack:plugin:graph:assemble', task ':x-pack:plugin:graph:copyCheckstyleConf', task ':x-pack:plugin:graph:checkstyleMain', task ':x-pack:plugin:graph:compileTestJava', task ':x-pack:plugin:graph:processTestResources', task ':x-pack:plugin:graph:testClasses', task ':x-pack:plugin:graph:checkstyleTest', task ':x-pack:plugin:graph:forbiddenApisMain', task ':x-pack:plugin:graph:forbiddenApisTest', task ':x-pack:plugin:graph:forbiddenApis', task ':x-pack:plugin:graph:copyRestSpec', task ':x-pack:plugin:graph:checkstyle', task ':x-pack:plugin:graph:dependencyLicenses', task ':x-pack:plugin:graph:filepermissions', task ':x-pack:plugin:graph:forbiddenPatterns', task ':x-pack:plugin:graph:jarHell', task ':x-pack:plugin:graph:licenseHeaders', task ':x-pack:plugin:graph:loggerUsageCheck', task ':x-pack:plugin:graph:namingConventions', task ':x-pack:plugin:graph:thirdPartyAudit', task ':x-pack:plugin:graph:precommit', task ':x-pack:plugin:graph:test', task ':x-pack:plugin:graph:integTestCluster#init', task ':x-pack:plugin:graph:integTestRunner', task ':x-pack:plugin:graph:integTest', task ':x-pack:plugin:graph:extractJar', task ':x-pack:plugin:graph:verifyJarLicense', task ':x-pack:plugin:graph:verifyJarNotice', task ':x-pack:plugin:graph:extractJavadocJar', task ':x-pack:plugin:graph:verifyJavadocJarLicense', task ':x-pack:plugin:graph:verifyJavadocJarNotice', task ':x-pack:plugin:graph:extractSourcesJar', task ':x-pack:plugin:graph:verifySourcesJarLicense', task ':x-pack:plugin:graph:verifySourcesJarNotice', task ':x-pack:plugin:graph:check', task ':x-pack:plugin:graph:build', task ':x-pack:plugin:logstash:generatePomFileForZipPublication', task ':x-pack:plugin:logstash:generatePomFileForZipRealPublication', task ':x-pack:plugin:logstash:javadoc', task ':x-pack:plugin:logstash:javadocJar', task ':x-pack:plugin:logstash:sourcesJar', task ':x-pack:plugin:logstash:assemble', task ':x-pack:plugin:logstash:copyCheckstyleConf', task ':x-pack:plugin:logstash:checkstyleMain', task ':x-pack:plugin:logstash:compileTestJava', task ':x-pack:plugin:logstash:processTestResources', task ':x-pack:plugin:logstash:testClasses', task ':x-pack:plugin:logstash:checkstyleTest', task ':x-pack:plugin:logstash:forbiddenApisMain', task ':x-pack:plugin:logstash:forbiddenApisTest', task ':x-pack:plugin:logstash:forbiddenApis', task ':x-pack:plugin:logstash:copyRestSpec', task ':x-pack:plugin:logstash:checkstyle', task ':x-pack:plugin:logstash:dependencyLicenses', task ':x-pack:plugin:logstash:filepermissions', task ':x-pack:plugin:logstash:forbiddenPatterns', task ':x-pack:plugin:logstash:jarHell', task ':x-pack:plugin:logstash:licenseHeaders', task ':x-pack:plugin:logstash:loggerUsageCheck', task ':x-pack:plugin:logstash:namingConventions', task ':x-pack:plugin:logstash:thirdPartyAudit', task ':x-pack:plugin:logstash:precommit', task ':x-pack:plugin:logstash:test', task ':x-pack:plugin:logstash:integTestCluster#init', task ':x-pack:plugin:logstash:integTestRunner', task ':x-pack:plugin:logstash:integTest', task ':x-pack:plugin:logstash:extractJar', task ':x-pack:plugin:logstash:verifyJarLicense', task ':x-pack:plugin:logstash:verifyJarNotice', task ':x-pack:plugin:logstash:extractJavadocJar', task ':x-pack:plugin:logstash:verifyJavadocJarLicense', task ':x-pack:plugin:logstash:verifyJavadocJarNotice', task ':x-pack:plugin:logstash:extractSourcesJar', task ':x-pack:plugin:logstash:verifySourcesJarLicense', task ':x-pack:plugin:logstash:verifySourcesJarNotice', task ':x-pack:plugin:logstash:check', task ':x-pack:plugin:logstash:build', task ':x-pack:plugin:ml:generatePomFileForZipPublication', task ':x-pack:plugin:ml:generatePomFileForZipRealPublication', task ':x-pack:plugin:ml:javadoc', task ':x-pack:plugin:ml:javadocJar', task ':x-pack:plugin:ml:sourcesJar', task ':x-pack:plugin:ml:assemble', task ':x-pack:plugin:ml:copyCheckstyleConf', task ':x-pack:plugin:ml:checkstyleMain', task ':x-pack:plugin:security:compileTestJava', task ':x-pack:plugin:security:processTestResources', task ':x-pack:plugin:security:testClasses', task ':x-pack:plugin:security:testJar', task ':x-pack:plugin:ml:compileTestJava', task ':x-pack:plugin:ml:processTestResources', task ':x-pack:plugin:ml:testClasses', task ':x-pack:plugin:ml:checkstyleTest', task ':x-pack:plugin:ml:forbiddenApisMain', task ':x-pack:plugin:ml:forbiddenApisTest', task ':x-pack:plugin:ml:forbiddenApis', task ':x-pack:plugin:ml:copyRestSpec', task ':x-pack:plugin:ml:checkstyle', task ':x-pack:plugin:ml:dependencyLicenses', task ':x-pack:plugin:ml:filepermissions', task ':x-pack:plugin:ml:forbiddenPatterns', task ':x-pack:plugin:ml:jarHell', task ':x-pack:plugin:ml:licenseHeaders', task ':x-pack:plugin:ml:loggerUsageCheck', task ':x-pack:plugin:ml:namingConventions', task ':x-pack:plugin:ml:thirdPartyAudit', task ':x-pack:plugin:ml:precommit', task ':x-pack:plugin:monitoring:copyCheckstyleConf', task ':x-pack:plugin:monitoring:checkstyleMain', task ':x-pack:plugin:monitoring:compileTestJava', task ':x-pack:plugin:monitoring:processTestResources', task ':x-pack:plugin:monitoring:testClasses', task ':x-pack:plugin:monitoring:checkstyleTest', task ':x-pack:plugin:monitoring:checkstyle', task ':x-pack:plugin:monitoring:dependencyLicenses', task ':x-pack:plugin:monitoring:filepermissions', task ':x-pack:plugin:monitoring:forbiddenApisMain', task ':x-pack:plugin:monitoring:forbiddenApisTest', task ':x-pack:plugin:monitoring:forbiddenApis', task ':x-pack:plugin:monitoring:forbiddenPatterns', task ':x-pack:plugin:monitoring:jarHell', task ':x-pack:plugin:monitoring:licenseHeaders', task ':x-pack:plugin:monitoring:loggerUsageCheck', task ':x-pack:plugin:monitoring:namingConventions', task ':x-pack:plugin:monitoring:thirdPartyAudit', task ':x-pack:plugin:monitoring:precommit', task ':x-pack:plugin:watcher:copyCheckstyleConf', task ':x-pack:plugin:watcher:checkstyleMain', task ':x-pack:plugin:watcher:compileTestJava', task ':x-pack:plugin:watcher:processTestResources', task ':x-pack:plugin:watcher:testClasses', task ':x-pack:plugin:watcher:checkstyleTest', task ':x-pack:plugin:watcher:checkstyle', task ':x-pack:plugin:watcher:dependencyLicenses', task ':x-pack:plugin:watcher:filepermissions', task ':x-pack:plugin:watcher:forbiddenApisMain', task ':x-pack:plugin:watcher:forbiddenApisTest', task ':x-pack:plugin:watcher:forbiddenApis', task ':x-pack:plugin:watcher:forbiddenPatterns', task ':x-pack:plugin:watcher:jarHell', task ':x-pack:plugin:watcher:licenseHeaders', task ':x-pack:plugin:watcher:loggerUsageCheck', task ':x-pack:plugin:watcher:namingConventions', task ':x-pack:plugin:watcher:thirdPartyAudit', task ':x-pack:plugin:watcher:precommit', task ':x-pack:plugin:watcher:test', task ':x-pack:plugin:monitoring:test', task ':x-pack:plugin:security:copyCheckstyleConf', task ':x-pack:plugin:security:checkstyleMain', task ':x-pack:plugin:security:checkstyleTest', task ':x-pack:plugin:security:checkstyle', task ':x-pack:plugin:security:dependencyLicenses', task ':x-pack:plugin:security:filepermissions', task ':x-pack:plugin:security:forbiddenApisMain', task ':x-pack:plugin:security:forbiddenApisTest', task ':x-pack:plugin:security:forbiddenApis', task ':x-pack:plugin:security:forbiddenPatterns', task ':x-pack:plugin:security:jarHell', task ':x-pack:plugin:security:licenseHeaders', task ':x-pack:plugin:security:loggerUsageCheck', task ':x-pack:plugin:security:namingConventions', task ':x-pack:plugin:security:thirdPartyAudit', task ':x-pack:plugin:security:precommit', task ':x-pack:plugin:sql:sql-proto:copyCheckstyleConf', task ':x-pack:plugin:sql:sql-proto:checkstyleMain', task ':x-pack:plugin:sql:sql-proto:compileTestJava', task ':x-pack:plugin:sql:sql-proto:processTestResources', task ':x-pack:plugin:sql:sql-proto:testClasses', task ':x-pack:plugin:sql:sql-proto:checkstyleTest', task ':x-pack:plugin:sql:sql-proto:checkstyle', task ':x-pack:plugin:sql:sql-proto:dependencyLicenses', task ':x-pack:plugin:sql:sql-proto:filepermissions', task ':x-pack:plugin:sql:sql-proto:forbiddenApisMain', task ':x-pack:plugin:sql:sql-proto:forbiddenApisTest', task ':x-pack:plugin:sql:sql-proto:forbiddenApis', task ':x-pack:plugin:sql:sql-proto:forbiddenPatterns', task ':x-pack:plugin:sql:sql-proto:jarHell', task ':x-pack:plugin:sql:sql-proto:licenseHeaders', task ':x-pack:plugin:sql:sql-proto:loggerUsageCheck', task ':x-pack:plugin:sql:sql-proto:namingConventions', task ':x-pack:plugin:sql:sql-proto:thirdPartyAudit', task ':x-pack:plugin:sql:sql-proto:precommit', task ':x-pack:plugin:sql:sql-proto:test', task ':x-pack:plugin:security:test', task ':x-pack:plugin:ml:test', task ':x-pack:plugin:security:copyRestSpec', task ':x-pack:plugin:monitoring:copyRestSpec', task ':x-pack:plugin:watcher:copyRestSpec', task ':x-pack:plugin:watcher:integTestCluster#init', task ':x-pack:plugin:watcher:integTestRunner', task ':x-pack:plugin:watcher:integTest', task ':x-pack:plugin:monitoring:integTestCluster#init', task ':x-pack:plugin:monitoring:integTestRunner', task ':x-pack:plugin:monitoring:integTest', task ':x-pack:plugin:security:integTestCluster#init', task ':x-pack:plugin:security:integTestRunner', task ':x-pack:plugin:security:integTest', task ':x-pack:plugin:ml:integTestCluster#init', task ':x-pack:plugin:ml:integTestRunner', task ':x-pack:plugin:ml:integTest', task ':x-pack:plugin:ml:internalClusterTest', task ':x-pack:plugin:ml:extractJar', task ':x-pack:plugin:ml:verifyJarLicense', task ':x-pack:plugin:ml:verifyJarNotice', task ':x-pack:plugin:ml:extractJavadocJar', task ':x-pack:plugin:ml:verifyJavadocJarLicense', task ':x-pack:plugin:ml:verifyJavadocJarNotice', task ':x-pack:plugin:ml:extractSourcesJar', task ':x-pack:plugin:ml:verifySourcesJarLicense', task ':x-pack:plugin:ml:verifySourcesJarNotice', task ':x-pack:plugin:ml:testJar', task ':x-pack:plugin:ml:extractTestJar', task ':x-pack:plugin:ml:verifyTestJarLicense', task ':x-pack:plugin:ml:verifyTestJarNotice', task ':x-pack:plugin:ml:check', task ':x-pack:plugin:ml:build', task ':x-pack:plugin:monitoring:generatePomFileForZipPublication', task ':x-pack:plugin:monitoring:generatePomFileForZipRealPublication', task ':x-pack:plugin:monitoring:javadoc', task ':x-pack:plugin:monitoring:javadocJar', task ':x-pack:plugin:monitoring:sourcesJar', task ':x-pack:plugin:monitoring:assemble', task ':x-pack:plugin:monitoring:internalClusterTest', task ':x-pack:plugin:monitoring:extractJar', task ':x-pack:plugin:monitoring:verifyJarLicense', task ':x-pack:plugin:monitoring:verifyJarNotice', task ':x-pack:plugin:monitoring:extractJavadocJar', task ':x-pack:plugin:monitoring:verifyJavadocJarLicense', task ':x-pack:plugin:monitoring:verifyJavadocJarNotice', task ':x-pack:plugin:monitoring:extractSourcesJar', task ':x-pack:plugin:monitoring:verifySourcesJarLicense', task ':x-pack:plugin:monitoring:verifySourcesJarNotice', task ':x-pack:plugin:monitoring:testJar', task ':x-pack:plugin:monitoring:extractTestJar', task ':x-pack:plugin:monitoring:verifyTestJarLicense', task ':x-pack:plugin:monitoring:verifyTestJarNotice', task ':x-pack:plugin:monitoring:check', task ':x-pack:plugin:monitoring:build', task ':x-pack:plugin:rollup:generatePomFileForZipPublication', task ':x-pack:plugin:rollup:generatePomFileForZipRealPublication', task ':x-pack:plugin:rollup:javadoc', task ':x-pack:plugin:rollup:javadocJar', task ':x-pack:plugin:rollup:sourcesJar', task ':x-pack:plugin:rollup:assemble', task ':x-pack:plugin:rollup:copyCheckstyleConf', task ':x-pack:plugin:rollup:checkstyleMain', task ':x-pack:plugin:rollup:compileTestJava', task ':x-pack:plugin:rollup:processTestResources', task ':x-pack:plugin:rollup:testClasses', task ':x-pack:plugin:rollup:checkstyleTest', task ':x-pack:plugin:rollup:forbiddenApisMain', task ':x-pack:plugin:rollup:forbiddenApisTest', task ':x-pack:plugin:rollup:forbiddenApis', task ':x-pack:plugin:rollup:copyRestSpec', task ':x-pack:plugin:rollup:checkstyle', task ':x-pack:plugin:rollup:dependencyLicenses', task ':x-pack:plugin:rollup:filepermissions', task ':x-pack:plugin:rollup:forbiddenPatterns', task ':x-pack:plugin:rollup:jarHell', task ':x-pack:plugin:rollup:licenseHeaders', task ':x-pack:plugin:rollup:loggerUsageCheck', task ':x-pack:plugin:rollup:namingConventions', task ':x-pack:plugin:rollup:thirdPartyAudit', task ':x-pack:plugin:rollup:precommit', task ':x-pack:plugin:rollup:test', task ':x-pack:plugin:rollup:integTestCluster#init', task ':x-pack:plugin:rollup:integTestRunner', task ':x-pack:plugin:rollup:integTest', task ':x-pack:plugin:rollup:internalClusterTest', task ':x-pack:plugin:rollup:extractJar', task ':x-pack:plugin:rollup:verifyJarLicense', task ':x-pack:plugin:rollup:verifyJarNotice', task ':x-pack:plugin:rollup:extractJavadocJar', task ':x-pack:plugin:rollup:verifyJavadocJarLicense', task ':x-pack:plugin:rollup:verifyJavadocJarNotice', task ':x-pack:plugin:rollup:extractSourcesJar', task ':x-pack:plugin:rollup:verifySourcesJarLicense', task ':x-pack:plugin:rollup:verifySourcesJarNotice', task ':x-pack:plugin:rollup:check', task ':x-pack:plugin:rollup:build', task ':x-pack:plugin:security:generatePomFileForZipPublication', task ':x-pack:plugin:security:generatePomFileForZipRealPublication', task ':x-pack:plugin:security:javadoc', task ':x-pack:plugin:security:javadocJar', task ':x-pack:plugin:security:sourcesJar', task ':x-pack:plugin:security:assemble', task ':x-pack:plugin:security:extractJar', task ':x-pack:plugin:security:verifyJarLicense', task ':x-pack:plugin:security:verifyJarNotice', task ':x-pack:plugin:security:extractJavadocJar', task ':x-pack:plugin:security:verifyJavadocJarLicense', task ':x-pack:plugin:security:verifyJavadocJarNotice', task ':x-pack:plugin:security:extractSourcesJar', task ':x-pack:plugin:security:verifySourcesJarLicense', task ':x-pack:plugin:security:verifySourcesJarNotice', task ':x-pack:plugin:security:extractTestJar', task ':x-pack:plugin:security:verifyTestJarLicense', task ':x-pack:plugin:security:verifyTestJarNotice', task ':x-pack:plugin:security:check', task ':x-pack:plugin:security:build', task ':x-pack:plugin:sql:generatePomFileForZipPublication', task ':x-pack:plugin:sql:generatePomFileForZipRealPublication', task ':x-pack:plugin:sql:sql-proto:javadoc', task ':x-pack:plugin:sql:javadoc', task ':x-pack:plugin:sql:javadocJar', task ':x-pack:plugin:sql:sourcesJar', task ':x-pack:plugin:sql:assemble', task ':x-pack:plugin:sql:copyCheckstyleConf', task ':x-pack:plugin:sql:checkstyleMain', task ':x-pack:plugin:sql:compileTestJava', task ':x-pack:plugin:sql:processTestResources', task ':x-pack:plugin:sql:testClasses', task ':x-pack:plugin:sql:checkstyleTest', task ':x-pack:plugin:sql:forbiddenApisMain', task ':x-pack:plugin:sql:forbiddenApisTest', task ':x-pack:plugin:sql:forbiddenApis', task ':x-pack:plugin:sql:copyRestSpec', task ':x-pack:plugin:sql:checkstyle', task ':x-pack:plugin:sql:dependencyLicenses', task ':x-pack:plugin:sql:filepermissions', task ':x-pack:plugin:sql:forbiddenPatterns', task ':x-pack:plugin:sql:jarHell', task ':x-pack:plugin:sql:licenseHeaders', task ':x-pack:plugin:sql:loggerUsageCheck', task ':x-pack:plugin:sql:namingConventions', task ':x-pack:plugin:sql:thirdPartyAudit', task ':x-pack:plugin:sql:precommit', task ':x-pack:plugin:sql:sql-cli:copyCheckstyleConf', task ':x-pack:plugin:sql:sql-cli:checkstyleMain', task ':x-pack:plugin:sql:sql-cli:compileTestJava', task ':x-pack:plugin:sql:sql-cli:processTestResources', task ':x-pack:plugin:sql:sql-cli:testClasses', task ':x-pack:plugin:sql:sql-cli:checkstyleTest', task ':x-pack:plugin:sql:sql-cli:checkstyle', task ':x-pack:plugin:sql:sql-cli:dependencyLicenses', task ':x-pack:plugin:sql:sql-cli:filepermissions', task ':x-pack:plugin:sql:sql-cli:forbiddenApisMain', task ':x-pack:plugin:sql:sql-cli:forbiddenApisTest', task ':x-pack:plugin:sql:sql-cli:forbiddenApis', task ':x-pack:plugin:sql:sql-cli:forbiddenPatterns', task ':x-pack:plugin:sql:sql-cli:jarHell', task ':x-pack:plugin:sql:sql-cli:licenseHeaders', task ':x-pack:plugin:sql:sql-cli:loggerUsageCheck', task ':x-pack:plugin:sql:sql-cli:namingConventions', task ':x-pack:plugin:sql:sql-cli:thirdPartyAudit', task ':x-pack:plugin:sql:sql-cli:precommit', task ':x-pack:plugin:sql:sql-shared-client:copyCheckstyleConf', task ':x-pack:plugin:sql:sql-shared-client:checkstyleMain', task ':x-pack:plugin:sql:sql-shared-client:compileTestJava', task ':x-pack:plugin:sql:sql-shared-client:processTestResources', task ':x-pack:plugin:sql:sql-shared-client:testClasses', task ':x-pack:plugin:sql:sql-shared-client:checkstyleTest', task ':x-pack:plugin:sql:sql-shared-client:checkstyle', task ':x-pack:plugin:sql:sql-shared-client:dependencyLicenses', task ':x-pack:plugin:sql:sql-shared-client:filepermissions', task ':x-pack:plugin:sql:sql-shared-client:forbiddenApisMain', task ':x-pack:plugin:sql:sql-shared-client:forbiddenApisTest', task ':x-pack:plugin:sql:sql-shared-client:forbiddenApis', task ':x-pack:plugin:sql:sql-shared-client:forbiddenPatterns', task ':x-pack:plugin:sql:sql-shared-client:jarHell', task ':x-pack:plugin:sql:sql-shared-client:licenseHeaders', task ':x-pack:plugin:sql:sql-shared-client:loggerUsageCheck', task ':x-pack:plugin:sql:sql-shared-client:namingConventions', task ':x-pack:plugin:sql:sql-shared-client:thirdPartyAudit', task ':x-pack:plugin:sql:sql-shared-client:precommit', task ':x-pack:plugin:sql:sql-shared-client:test', task ':x-pack:plugin:sql:sql-cli:test', task ':x-pack:plugin:sql:test', task ':x-pack:plugin:sql:integTestCluster#init', task ':x-pack:plugin:sql:integTestRunner', task ':x-pack:plugin:sql:integTest', task ':x-pack:plugin:sql:extractJar', task ':x-pack:plugin:sql:verifyJarLicense', task ':x-pack:plugin:sql:verifyJarNotice', task ':x-pack:plugin:sql:extractJavadocJar', task ':x-pack:plugin:sql:verifyJavadocJarLicense', task ':x-pack:plugin:sql:verifyJavadocJarNotice', task ':x-pack:plugin:sql:extractSourcesJar', task ':x-pack:plugin:sql:verifySourcesJarLicense', task ':x-pack:plugin:sql:verifySourcesJarNotice', task ':x-pack:plugin:sql:check', task ':x-pack:plugin:sql:build', task ':x-pack:plugin:upgrade:generatePomFileForZipPublication', task ':x-pack:plugin:upgrade:generatePomFileForZipRealPublication', task ':x-pack:plugin:upgrade:javadoc', task ':x-pack:plugin:upgrade:javadocJar', task ':x-pack:plugin:upgrade:sourcesJar', task ':x-pack:plugin:upgrade:assemble', task ':x-pack:plugin:upgrade:copyCheckstyleConf', task ':x-pack:plugin:upgrade:checkstyleMain', task ':x-pack:plugin:upgrade:compileTestJava', task ':x-pack:plugin:upgrade:processTestResources', task ':x-pack:plugin:upgrade:testClasses', task ':x-pack:plugin:upgrade:checkstyleTest', task ':x-pack:plugin:upgrade:forbiddenApisMain', task ':x-pack:plugin:upgrade:forbiddenApisTest', task ':x-pack:plugin:upgrade:forbiddenApis', task ':x-pack:plugin:upgrade:copyRestSpec', task ':x-pack:plugin:upgrade:checkstyle', task ':x-pack:plugin:upgrade:dependencyLicenses', task ':x-pack:plugin:upgrade:filepermissions', task ':x-pack:plugin:upgrade:forbiddenPatterns', task ':x-pack:plugin:upgrade:jarHell', task ':x-pack:plugin:upgrade:licenseHeaders', task ':x-pack:plugin:upgrade:loggerUsageCheck', task ':x-pack:plugin:upgrade:namingConventions', task ':x-pack:plugin:upgrade:thirdPartyAudit', task ':x-pack:plugin:upgrade:precommit', task ':x-pack:plugin:upgrade:test', task ':x-pack:plugin:upgrade:integTestCluster#init', task ':x-pack:plugin:upgrade:integTestRunner', task ':x-pack:plugin:upgrade:integTest', task ':x-pack:plugin:upgrade:internalClusterTest', task ':x-pack:plugin:upgrade:extractJar', task ':x-pack:plugin:upgrade:verifyJarLicense', task ':x-pack:plugin:upgrade:verifyJarNotice', task ':x-pack:plugin:upgrade:extractJavadocJar', task ':x-pack:plugin:upgrade:verifyJavadocJarLicense', task ':x-pack:plugin:upgrade:verifyJavadocJarNotice', task ':x-pack:plugin:upgrade:extractSourcesJar', task ':x-pack:plugin:upgrade:verifySourcesJarLicense', task ':x-pack:plugin:upgrade:verifySourcesJarNotice', task ':x-pack:plugin:upgrade:check', task ':x-pack:plugin:upgrade:build', task ':x-pack:plugin:watcher:generatePomFileForZipPublication', task ':x-pack:plugin:watcher:generatePomFileForZipRealPublication', task ':x-pack:plugin:watcher:javadoc', task ':x-pack:plugin:watcher:javadocJar', task ':x-pack:plugin:watcher:sourcesJar', task ':x-pack:plugin:watcher:assemble', task ':x-pack:plugin:watcher:extractJar', task ':x-pack:plugin:watcher:verifyJarLicense', task ':x-pack:plugin:watcher:verifyJarNotice', task ':x-pack:plugin:watcher:extractJavadocJar', task ':x-pack:plugin:watcher:verifyJavadocJarLicense', task ':x-pack:plugin:watcher:verifyJavadocJarNotice', task ':x-pack:plugin:watcher:extractSourcesJar', task ':x-pack:plugin:watcher:verifySourcesJarLicense', task ':x-pack:plugin:watcher:verifySourcesJarNotice', task ':x-pack:plugin:watcher:check', task ':x-pack:plugin:watcher:build', task ':x-pack:qa:audit-tests:copyCheckstyleConf', task ':x-pack:qa:audit-tests:copyXPackPluginProps', task ':x-pack:qa:audit-tests:compileTestJava', task ':x-pack:qa:audit-tests:processTestResources', task ':x-pack:qa:audit-tests:testClasses', task ':x-pack:qa:audit-tests:checkstyleTest', task ':x-pack:qa:audit-tests:forbiddenApisTest', task ':x-pack:qa:audit-tests:forbiddenApis', task ':x-pack:qa:audit-tests:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:audit-tests:integTestCluster#clean', task ':x-pack:qa:audit-tests:integTestCluster#createCwd', task ':x-pack:qa:audit-tests:integTestCluster#checkPrevious', task ':x-pack:qa:audit-tests:integTestCluster#stopPrevious', task ':x-pack:qa:audit-tests:integTestCluster#extract', task ':x-pack:qa:audit-tests:integTestCluster#configure', task ':x-pack:qa:audit-tests:integTestCluster#setupDummyUser', task ':x-pack:qa:audit-tests:integTestCluster#start', task ':x-pack:qa:audit-tests:copyRestSpec', task ':x-pack:qa:audit-tests:checkstyle', task ':x-pack:qa:audit-tests:filepermissions', task ':x-pack:qa:audit-tests:forbiddenPatterns', task ':x-pack:qa:audit-tests:jarHell', task ':x-pack:qa:audit-tests:licenseHeaders', task ':x-pack:qa:audit-tests:loggerUsageCheck', task ':x-pack:qa:audit-tests:namingConventions', task ':x-pack:qa:audit-tests:thirdPartyAudit', task ':x-pack:qa:audit-tests:precommit', task ':x-pack:qa:audit-tests:integTestCluster#init', task ':x-pack:qa:audit-tests:integTestCluster#wait', task ':x-pack:qa:audit-tests:integTestRunner', task ':x-pack:qa:audit-tests:integTestCluster#stop', task ':x-pack:qa:audit-tests:integTest', task ':x-pack:qa:audit-tests:check', task ':x-pack:qa:audit-tests:build', task ':x-pack:qa:core-rest-tests-with-security:copyCheckstyleConf', task ':x-pack:qa:core-rest-tests-with-security:compileTestJava', task ':x-pack:qa:core-rest-tests-with-security:processTestResources', task ':x-pack:qa:core-rest-tests-with-security:testClasses', task ':x-pack:qa:core-rest-tests-with-security:checkstyleTest', task ':x-pack:qa:core-rest-tests-with-security:forbiddenApisTest', task ':x-pack:qa:core-rest-tests-with-security:forbiddenApis', task ':x-pack:qa:core-rest-tests-with-security:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:core-rest-tests-with-security:integTestCluster#clean', task ':x-pack:qa:core-rest-tests-with-security:integTestCluster#createCwd', task ':x-pack:qa:core-rest-tests-with-security:integTestCluster#checkPrevious', task ':x-pack:qa:core-rest-tests-with-security:integTestCluster#stopPrevious', task ':x-pack:qa:core-rest-tests-with-security:integTestCluster#extract', task ':x-pack:qa:core-rest-tests-with-security:integTestCluster#configure', task ':x-pack:qa:core-rest-tests-with-security:integTestCluster#setupDummyUser', task ':x-pack:qa:core-rest-tests-with-security:integTestCluster#start', task ':x-pack:qa:core-rest-tests-with-security:copyRestSpec', task ':x-pack:qa:core-rest-tests-with-security:checkstyle', task ':x-pack:qa:core-rest-tests-with-security:filepermissions', task ':x-pack:qa:core-rest-tests-with-security:forbiddenPatterns', task ':x-pack:qa:core-rest-tests-with-security:jarHell', task ':x-pack:qa:core-rest-tests-with-security:licenseHeaders', task ':x-pack:qa:core-rest-tests-with-security:loggerUsageCheck', task ':x-pack:qa:core-rest-tests-with-security:namingConventions', task ':x-pack:qa:core-rest-tests-with-security:thirdPartyAudit', task ':x-pack:qa:core-rest-tests-with-security:precommit', task ':x-pack:qa:core-rest-tests-with-security:integTestCluster#init', task ':x-pack:qa:core-rest-tests-with-security:integTestCluster#wait', task ':x-pack:qa:core-rest-tests-with-security:integTestRunner', task ':x-pack:qa:core-rest-tests-with-security:integTestCluster#stop', task ':x-pack:qa:core-rest-tests-with-security:integTest', task ':x-pack:qa:core-rest-tests-with-security:check', task ':x-pack:qa:core-rest-tests-with-security:build', task ':x-pack:qa:full-cluster-restart:compileJava', task ':x-pack:qa:full-cluster-restart:processResources', task ':x-pack:qa:full-cluster-restart:classes', task ':x-pack:qa:full-cluster-restart:copyCheckstyleConf', task ':x-pack:qa:full-cluster-restart:checkstyleMain', task ':x-pack:qa:full-cluster-restart:compileTestJava', task ':x-pack:qa:full-cluster-restart:processTestResources', task ':x-pack:qa:full-cluster-restart:testClasses', task ':x-pack:qa:full-cluster-restart:checkstyleTest', task ':x-pack:qa:full-cluster-restart:forbiddenApisMain', task ':x-pack:qa:full-cluster-restart:forbiddenApisTest', task ':x-pack:qa:full-cluster-restart:forbiddenApis', task ':x-pack:qa:full-cluster-restart:checkstyle', task ':x-pack:qa:full-cluster-restart:dependencyLicenses', task ':x-pack:qa:full-cluster-restart:filepermissions', task ':x-pack:qa:full-cluster-restart:forbiddenPatterns', task ':x-pack:qa:full-cluster-restart:jarHell', task ':x-pack:qa:full-cluster-restart:licenseHeaders', task ':x-pack:qa:full-cluster-restart:loggerUsageCheck', task ':x-pack:qa:full-cluster-restart:namingConventions', task ':x-pack:qa:full-cluster-restart:thirdPartyAudit', task ':x-pack:qa:full-cluster-restart:precommit', task ':x-pack:qa:full-cluster-restart:test', task ':x-pack:qa:full-cluster-restart:jar', task ':x-pack:qa:full-cluster-restart:extractJar', task ':x-pack:qa:full-cluster-restart:verifyJarLicense', task ':x-pack:qa:full-cluster-restart:verifyJarNotice', task ':x-pack:qa:full-cluster-restart:javadoc', task ':x-pack:qa:full-cluster-restart:javadocJar', task ':x-pack:qa:full-cluster-restart:extractJavadocJar', task ':x-pack:qa:full-cluster-restart:verifyJavadocJarLicense', task ':x-pack:qa:full-cluster-restart:verifyJavadocJarNotice', task ':x-pack:qa:full-cluster-restart:sourcesJar', task ':x-pack:qa:full-cluster-restart:extractSourcesJar', task ':x-pack:qa:full-cluster-restart:verifySourcesJarLicense', task ':x-pack:qa:full-cluster-restart:verifySourcesJarNotice', task ':x-pack:qa:full-cluster-restart:check', task ':x-pack:qa:full-cluster-restart:build', task ':x-pack:qa:ml-basic-multi-node:copyCheckstyleConf', task ':x-pack:qa:ml-basic-multi-node:compileTestJava', task ':x-pack:qa:ml-basic-multi-node:processTestResources', task ':x-pack:qa:ml-basic-multi-node:testClasses', task ':x-pack:qa:ml-basic-multi-node:checkstyleTest', task ':x-pack:qa:ml-basic-multi-node:forbiddenApisTest', task ':x-pack:qa:ml-basic-multi-node:forbiddenApis', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node0.clean', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node0.createCwd', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node0.checkPrevious', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node0.stopPrevious', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node0.extract', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node0.configure', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node0.start', task ':x-pack:qa:ml-basic-multi-node:copyRestSpec', task ':x-pack:qa:ml-basic-multi-node:checkstyle', task ':x-pack:qa:ml-basic-multi-node:filepermissions', task ':x-pack:qa:ml-basic-multi-node:forbiddenPatterns', task ':x-pack:qa:ml-basic-multi-node:jarHell', task ':x-pack:qa:ml-basic-multi-node:licenseHeaders', task ':x-pack:qa:ml-basic-multi-node:loggerUsageCheck', task ':x-pack:qa:ml-basic-multi-node:namingConventions', task ':x-pack:qa:ml-basic-multi-node:thirdPartyAudit', task ':x-pack:qa:ml-basic-multi-node:precommit', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#init', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node1.clean', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node1.createCwd', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node1.checkPrevious', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node1.stopPrevious', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node1.extract', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node1.configure', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node1.start', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node2.clean', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node2.createCwd', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node2.checkPrevious', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node2.stopPrevious', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node2.extract', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node2.configure', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node2.start', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#wait', task ':x-pack:qa:ml-basic-multi-node:integTestRunner', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node2.stop', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node1.stop', task ':x-pack:qa:ml-basic-multi-node:integTestCluster#node0.stop', task ':x-pack:qa:ml-basic-multi-node:integTest', task ':x-pack:qa:ml-basic-multi-node:check', task ':x-pack:qa:ml-basic-multi-node:build', task ':x-pack:qa:ml-disabled:copyCheckstyleConf', task ':x-pack:qa:ml-disabled:compileTestJava', task ':x-pack:qa:ml-disabled:processTestResources', task ':x-pack:qa:ml-disabled:testClasses', task ':x-pack:qa:ml-disabled:checkstyleTest', task ':x-pack:qa:ml-disabled:forbiddenApisTest', task ':x-pack:qa:ml-disabled:forbiddenApis', task ':x-pack:qa:ml-disabled:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:ml-disabled:integTestCluster#clean', task ':x-pack:qa:ml-disabled:integTestCluster#createCwd', task ':x-pack:qa:ml-disabled:integTestCluster#checkPrevious', task ':x-pack:qa:ml-disabled:integTestCluster#stopPrevious', task ':x-pack:qa:ml-disabled:integTestCluster#extract', task ':x-pack:qa:ml-disabled:integTestCluster#configure', task ':x-pack:qa:ml-disabled:integTestCluster#start', task ':x-pack:qa:ml-disabled:copyRestSpec', task ':x-pack:qa:ml-disabled:checkstyle', task ':x-pack:qa:ml-disabled:filepermissions', task ':x-pack:qa:ml-disabled:forbiddenPatterns', task ':x-pack:qa:ml-disabled:jarHell', task ':x-pack:qa:ml-disabled:licenseHeaders', task ':x-pack:qa:ml-disabled:loggerUsageCheck', task ':x-pack:qa:ml-disabled:namingConventions', task ':x-pack:qa:ml-disabled:thirdPartyAudit', task ':x-pack:qa:ml-disabled:precommit', task ':x-pack:qa:ml-disabled:integTestCluster#init', task ':x-pack:qa:ml-disabled:integTestCluster#wait', task ':x-pack:qa:ml-disabled:integTestRunner', task ':x-pack:qa:ml-disabled:integTestCluster#stop', task ':x-pack:qa:ml-disabled:integTest', task ':x-pack:qa:ml-disabled:check', task ':x-pack:qa:ml-disabled:build', task ':x-pack:qa:ml-native-tests:copyCheckstyleConf', task ':x-pack:qa:ml-native-tests:compileTestJava', task ':x-pack:qa:ml-native-tests:createNodeKeyStore', task ':x-pack:qa:ml-native-tests:processTestResources', task ':x-pack:qa:ml-native-tests:testClasses', task ':x-pack:qa:ml-native-tests:checkstyleTest', task ':x-pack:qa:ml-native-tests:forbiddenApisTest', task ':x-pack:qa:ml-native-tests:forbiddenApis', task ':x-pack:qa:ml-native-tests:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:ml-native-tests:integTestCluster#clean', task ':x-pack:qa:ml-native-tests:integTestCluster#createCwd', task ':x-pack:qa:ml-native-tests:integTestCluster#checkPrevious', task ':x-pack:qa:ml-native-tests:integTestCluster#stopPrevious', task ':x-pack:qa:ml-native-tests:integTestCluster#extract', task ':x-pack:qa:ml-native-tests:integTestCluster#configure', task ':x-pack:qa:ml-native-tests:integTestCluster#createKeystore', task ':x-pack:qa:ml-native-tests:integTestCluster#addToKeystore#xpack.security.transport.ssl.keystore.secure_password', task ':x-pack:qa:ml-native-tests:integTestCluster#addToKeystore#bootstrap.password', task ':x-pack:qa:ml-native-tests:integTestCluster#extraConfig', task ':x-pack:qa:ml-native-tests:integTestCluster#setupDummyUser', task ':x-pack:qa:ml-native-tests:integTestCluster#start', task ':x-pack:qa:ml-native-tests:copyRestSpec', task ':x-pack:qa:ml-native-tests:checkstyle', task ':x-pack:qa:ml-native-tests:filepermissions', task ':x-pack:qa:ml-native-tests:forbiddenPatterns', task ':x-pack:qa:ml-native-tests:jarHell', task ':x-pack:qa:ml-native-tests:licenseHeaders', task ':x-pack:qa:ml-native-tests:loggerUsageCheck', task ':x-pack:qa:ml-native-tests:namingConventions', task ':x-pack:qa:ml-native-tests:thirdPartyAudit', task ':x-pack:qa:ml-native-tests:precommit', task ':x-pack:qa:ml-native-tests:integTestCluster#init', task ':x-pack:qa:ml-native-tests:integTestCluster#wait', task ':x-pack:qa:ml-native-tests:integTestRunner', task ':x-pack:qa:ml-native-tests:integTestCluster#stop', task ':x-pack:qa:ml-native-tests:integTest', task ':x-pack:qa:ml-native-tests:check', task ':x-pack:qa:ml-native-tests:build', task ':x-pack:qa:ml-no-bootstrap-tests:copyCheckstyleConf', task ':x-pack:qa:ml-no-bootstrap-tests:compileTestJava', task ':x-pack:qa:ml-no-bootstrap-tests:processTestResources', task ':x-pack:qa:ml-no-bootstrap-tests:testClasses', task ':x-pack:qa:ml-no-bootstrap-tests:checkstyleTest', task ':x-pack:qa:ml-no-bootstrap-tests:forbiddenApisTest', task ':x-pack:qa:ml-no-bootstrap-tests:forbiddenApis', task ':x-pack:qa:ml-no-bootstrap-tests:checkstyle', task ':x-pack:qa:ml-no-bootstrap-tests:filepermissions', task ':x-pack:qa:ml-no-bootstrap-tests:forbiddenPatterns', task ':x-pack:qa:ml-no-bootstrap-tests:jarHell', task ':x-pack:qa:ml-no-bootstrap-tests:licenseHeaders', task ':x-pack:qa:ml-no-bootstrap-tests:loggerUsageCheck', task ':x-pack:qa:ml-no-bootstrap-tests:namingConventions', task ':x-pack:qa:ml-no-bootstrap-tests:thirdPartyAudit', task ':x-pack:qa:ml-no-bootstrap-tests:precommit', task ':x-pack:qa:ml-no-bootstrap-tests:test', task ':x-pack:qa:ml-no-bootstrap-tests:check', task ':x-pack:qa:ml-no-bootstrap-tests:build', task ':x-pack:qa:ml-single-node-tests:copyCheckstyleConf', task ':x-pack:qa:ml-single-node-tests:compileTestJava', task ':x-pack:qa:ml-single-node-tests:processTestResources', task ':x-pack:qa:ml-single-node-tests:testClasses', task ':x-pack:qa:ml-single-node-tests:checkstyleTest', task ':x-pack:qa:ml-single-node-tests:forbiddenApisTest', task ':x-pack:qa:ml-single-node-tests:forbiddenApis', task ':x-pack:qa:ml-single-node-tests:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:ml-single-node-tests:integTestCluster#clean', task ':x-pack:qa:ml-single-node-tests:integTestCluster#createCwd', task ':x-pack:qa:ml-single-node-tests:integTestCluster#checkPrevious', task ':x-pack:qa:ml-single-node-tests:integTestCluster#stopPrevious', task ':x-pack:qa:ml-single-node-tests:integTestCluster#extract', task ':x-pack:qa:ml-single-node-tests:integTestCluster#configure', task ':x-pack:qa:ml-single-node-tests:integTestCluster#start', task ':x-pack:qa:ml-single-node-tests:copyRestSpec', task ':x-pack:qa:ml-single-node-tests:checkstyle', task ':x-pack:qa:ml-single-node-tests:filepermissions', task ':x-pack:qa:ml-single-node-tests:forbiddenPatterns', task ':x-pack:qa:ml-single-node-tests:jarHell', task ':x-pack:qa:ml-single-node-tests:licenseHeaders', task ':x-pack:qa:ml-single-node-tests:loggerUsageCheck', task ':x-pack:qa:ml-single-node-tests:namingConventions', task ':x-pack:qa:ml-single-node-tests:thirdPartyAudit', task ':x-pack:qa:ml-single-node-tests:precommit', task ':x-pack:qa:ml-single-node-tests:integTestCluster#init', task ':x-pack:qa:ml-single-node-tests:integTestCluster#wait', task ':x-pack:qa:ml-single-node-tests:integTestRunner', task ':x-pack:qa:ml-single-node-tests:integTestCluster#stop', task ':x-pack:qa:ml-single-node-tests:integTest', task ':x-pack:qa:ml-single-node-tests:check', task ':x-pack:qa:ml-single-node-tests:build', task ':x-pack:qa:multi-cluster-search-security:copyCheckstyleConf', task ':x-pack:qa:multi-cluster-search-security:compileTestJava', task ':x-pack:qa:multi-cluster-search-security:processTestResources', task ':x-pack:qa:multi-cluster-search-security:testClasses', task ':x-pack:qa:multi-cluster-search-security:checkstyleTest', task ':x-pack:qa:multi-cluster-search-security:forbiddenApisTest', task ':x-pack:qa:multi-cluster-search-security:forbiddenApis', task ':x-pack:qa:multi-cluster-search-security:copyRestSpec', task ':x-pack:qa:multi-cluster-search-security:checkstyle', task ':x-pack:qa:multi-cluster-search-security:filepermissions', task ':x-pack:qa:multi-cluster-search-security:forbiddenPatterns', task ':x-pack:qa:multi-cluster-search-security:jarHell', task ':x-pack:qa:multi-cluster-search-security:licenseHeaders', task ':x-pack:qa:multi-cluster-search-security:loggerUsageCheck', task ':x-pack:qa:multi-cluster-search-security:namingConventions', task ':x-pack:qa:multi-cluster-search-security:thirdPartyAudit', task ':x-pack:qa:multi-cluster-search-security:precommit', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#init', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node0.clean', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node0.createCwd', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node0.checkPrevious', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node0.stopPrevious', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node0.extract', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node0.configure', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node0.setupDummyUser', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node0.start', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node1.clean', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node1.createCwd', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node1.checkPrevious', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node1.stopPrevious', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node1.extract', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node1.configure', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node1.setupDummyUser', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node1.start', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#wait', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestRunner', task ':x-pack:qa:multi-cluster-search-security:mixedClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:multi-cluster-search-security:mixedClusterTestCluster#clean', task ':x-pack:qa:multi-cluster-search-security:mixedClusterTestCluster#createCwd', task ':x-pack:qa:multi-cluster-search-security:mixedClusterTestCluster#checkPrevious', task ':x-pack:qa:multi-cluster-search-security:mixedClusterTestCluster#stopPrevious', task ':x-pack:qa:multi-cluster-search-security:mixedClusterTestCluster#extract', task ':x-pack:qa:multi-cluster-search-security:mixedClusterTestCluster#configure', task ':x-pack:qa:multi-cluster-search-security:mixedClusterTestCluster#setupDummyUser', task ':x-pack:qa:multi-cluster-search-security:mixedClusterTestCluster#start', task ':x-pack:qa:multi-cluster-search-security:mixedClusterTestCluster#init', task ':x-pack:qa:multi-cluster-search-security:mixedClusterTestCluster#wait', task ':x-pack:qa:multi-cluster-search-security:mixedClusterTestRunner', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node0.stop', task ':x-pack:qa:multi-cluster-search-security:remoteClusterTestCluster#node1.stop', task ':x-pack:qa:multi-cluster-search-security:mixedClusterTestCluster#stop', task ':x-pack:qa:multi-cluster-search-security:mixedClusterTest', task ':x-pack:qa:multi-cluster-search-security:integTest', task ':x-pack:qa:multi-cluster-search-security:test', task ':x-pack:qa:multi-cluster-search-security:check', task ':x-pack:qa:multi-cluster-search-security:build', task ':x-pack:qa:multi-node:copyCheckstyleConf', task ':x-pack:qa:multi-node:compileTestJava', task ':x-pack:qa:multi-node:processTestResources', task ':x-pack:qa:multi-node:testClasses', task ':x-pack:qa:multi-node:checkstyleTest', task ':x-pack:qa:multi-node:forbiddenApisTest', task ':x-pack:qa:multi-node:forbiddenApis', task ':x-pack:qa:multi-node:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:multi-node:integTestCluster#node0.clean', task ':x-pack:qa:multi-node:integTestCluster#node0.createCwd', task ':x-pack:qa:multi-node:integTestCluster#node0.checkPrevious', task ':x-pack:qa:multi-node:integTestCluster#node0.stopPrevious', task ':x-pack:qa:multi-node:integTestCluster#node0.extract', task ':x-pack:qa:multi-node:integTestCluster#node0.configure', task ':x-pack:qa:multi-node:integTestCluster#node0.extraConfig', task ':x-pack:qa:multi-node:integTestCluster#node0.setup-test-user', task ':x-pack:qa:multi-node:integTestCluster#node0.setup-super-user', task ':x-pack:qa:multi-node:integTestCluster#node0.start', task ':x-pack:qa:multi-node:copyRestSpec', task ':x-pack:qa:multi-node:checkstyle', task ':x-pack:qa:multi-node:filepermissions', task ':x-pack:qa:multi-node:forbiddenPatterns', task ':x-pack:qa:multi-node:jarHell', task ':x-pack:qa:multi-node:licenseHeaders', task ':x-pack:qa:multi-node:loggerUsageCheck', task ':x-pack:qa:multi-node:namingConventions', task ':x-pack:qa:multi-node:thirdPartyAudit', task ':x-pack:qa:multi-node:precommit', task ':x-pack:qa:multi-node:integTestCluster#init', task ':x-pack:qa:multi-node:integTestCluster#node1.clean', task ':x-pack:qa:multi-node:integTestCluster#node1.createCwd', task ':x-pack:qa:multi-node:integTestCluster#node1.checkPrevious', task ':x-pack:qa:multi-node:integTestCluster#node1.stopPrevious', task ':x-pack:qa:multi-node:integTestCluster#node1.extract', task ':x-pack:qa:multi-node:integTestCluster#node1.configure', task ':x-pack:qa:multi-node:integTestCluster#node1.extraConfig', task ':x-pack:qa:multi-node:integTestCluster#node1.setup-test-user', task ':x-pack:qa:multi-node:integTestCluster#node1.setup-super-user', task ':x-pack:qa:multi-node:integTestCluster#node1.start', task ':x-pack:qa:multi-node:integTestCluster#wait', task ':x-pack:qa:multi-node:integTestRunner', task ':x-pack:qa:multi-node:integTestCluster#node1.stop', task ':x-pack:qa:multi-node:integTestCluster#node0.stop', task ':x-pack:qa:multi-node:integTest', task ':x-pack:qa:multi-node:check', task ':x-pack:qa:multi-node:build', task ':x-pack:qa:openldap-tests:copyCheckstyleConf', task ':x-pack:qa:openldap-tests:compileTestJava', task ':x-pack:qa:openldap-tests:processTestResources', task ':x-pack:qa:openldap-tests:testClasses', task ':x-pack:qa:openldap-tests:checkstyleTest', task ':x-pack:qa:openldap-tests:forbiddenApisTest', task ':x-pack:qa:openldap-tests:forbiddenApis', task ':x-pack:qa:openldap-tests:checkstyle', task ':x-pack:qa:openldap-tests:filepermissions', task ':x-pack:qa:openldap-tests:forbiddenPatterns', task ':x-pack:qa:openldap-tests:jarHell', task ':x-pack:qa:openldap-tests:licenseHeaders', task ':x-pack:qa:openldap-tests:loggerUsageCheck', task ':x-pack:qa:openldap-tests:namingConventions', task ':x-pack:qa:openldap-tests:thirdPartyAudit', task ':x-pack:qa:openldap-tests:precommit', task ':x-pack:qa:openldap-tests:test', task ':x-pack:qa:openldap-tests:check', task ':x-pack:qa:openldap-tests:build', task ':x-pack:qa:reindex-tests-with-security:copyCheckstyleConf', task ':x-pack:qa:reindex-tests-with-security:compileTestJava', task ':x-pack:qa:reindex-tests-with-security:processTestResources', task ':x-pack:qa:reindex-tests-with-security:testClasses', task ':x-pack:qa:reindex-tests-with-security:checkstyleTest', task ':x-pack:qa:reindex-tests-with-security:forbiddenApisTest', task ':x-pack:qa:reindex-tests-with-security:forbiddenApis', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#clean', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#createCwd', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#checkPrevious', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#stopPrevious', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#extract', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#configure', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#extraConfig', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#setupUser#test_admin', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#setupUser#powerful_user', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#setupUser#minimal_user', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#setupUser#readonly_user', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#setupUser#dest_only_user', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#setupUser#can_not_see_hidden_docs_user', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#setupUser#can_not_see_hidden_fields_user', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#start', task ':x-pack:qa:reindex-tests-with-security:copyRestSpec', task ':x-pack:qa:reindex-tests-with-security:checkstyle', task ':x-pack:qa:reindex-tests-with-security:filepermissions', task ':x-pack:qa:reindex-tests-with-security:forbiddenPatterns', task ':x-pack:qa:reindex-tests-with-security:jarHell', task ':x-pack:qa:reindex-tests-with-security:licenseHeaders', task ':x-pack:qa:reindex-tests-with-security:loggerUsageCheck', task ':x-pack:qa:reindex-tests-with-security:namingConventions', task ':x-pack:qa:reindex-tests-with-security:thirdPartyAudit', task ':x-pack:qa:reindex-tests-with-security:precommit', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#init', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#wait', task ':x-pack:qa:reindex-tests-with-security:integTestRunner', task ':x-pack:qa:reindex-tests-with-security:integTestCluster#stop', task ':x-pack:qa:reindex-tests-with-security:integTest', task ':x-pack:qa:reindex-tests-with-security:check', task ':x-pack:qa:reindex-tests-with-security:build', task ':x-pack:qa:rolling-upgrade:compileJava', task ':x-pack:qa:rolling-upgrade:processResources', task ':x-pack:qa:rolling-upgrade:classes', task ':x-pack:qa:rolling-upgrade:copyCheckstyleConf', task ':x-pack:qa:rolling-upgrade:checkstyleMain', task ':x-pack:qa:rolling-upgrade:compileTestJava', task ':x-pack:qa:rolling-upgrade:processTestResources', task ':x-pack:qa:rolling-upgrade:testClasses', task ':x-pack:qa:rolling-upgrade:checkstyleTest', task ':x-pack:qa:rolling-upgrade:forbiddenApisMain', task ':x-pack:qa:rolling-upgrade:forbiddenApisTest', task ':x-pack:qa:rolling-upgrade:forbiddenApis', task ':x-pack:qa:rolling-upgrade:checkstyle', task ':x-pack:qa:rolling-upgrade:dependencyLicenses', task ':x-pack:qa:rolling-upgrade:filepermissions', task ':x-pack:qa:rolling-upgrade:forbiddenPatterns', task ':x-pack:qa:rolling-upgrade:jarHell', task ':x-pack:qa:rolling-upgrade:licenseHeaders', task ':x-pack:qa:rolling-upgrade:loggerUsageCheck', task ':x-pack:qa:rolling-upgrade:namingConventions', task ':x-pack:qa:rolling-upgrade:thirdPartyAudit', task ':x-pack:qa:rolling-upgrade:precommit', task ':x-pack:qa:rolling-upgrade:test', task ':x-pack:qa:rolling-upgrade:jar', task ':x-pack:qa:rolling-upgrade:extractJar', task ':x-pack:qa:rolling-upgrade:verifyJarLicense', task ':x-pack:qa:rolling-upgrade:verifyJarNotice', task ':x-pack:qa:rolling-upgrade:javadoc', task ':x-pack:qa:rolling-upgrade:javadocJar', task ':x-pack:qa:rolling-upgrade:extractJavadocJar', task ':x-pack:qa:rolling-upgrade:verifyJavadocJarLicense', task ':x-pack:qa:rolling-upgrade:verifyJavadocJarNotice', task ':x-pack:qa:rolling-upgrade:sourcesJar', task ':x-pack:qa:rolling-upgrade:extractSourcesJar', task ':x-pack:qa:rolling-upgrade:verifySourcesJarLicense', task ':x-pack:qa:rolling-upgrade:verifySourcesJarNotice', task ':x-pack:qa:rolling-upgrade:check', task ':x-pack:qa:rolling-upgrade:build', task ':x-pack:qa:rolling-upgrade-basic:copyCheckstyleConf', task ':x-pack:qa:rolling-upgrade-basic:processTestResources', task ':x-pack:qa:rolling-upgrade-basic:copyXPackRestSpec', task ':x-pack:qa:rolling-upgrade-basic:copyXPackPluginProps', task ':x-pack:qa:rolling-upgrade-basic:compileTestJava', task ':x-pack:qa:rolling-upgrade-basic:testClasses', task ':x-pack:qa:rolling-upgrade-basic:checkstyleTest', task ':x-pack:qa:rolling-upgrade-basic:forbiddenApisTest', task ':x-pack:qa:rolling-upgrade-basic:forbiddenApis', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.clean', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.createCwd', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.checkPrevious', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.stopPrevious', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.extract', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.configure', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.start', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.clean', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.createCwd', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.checkPrevious', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.stopPrevious', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.extract', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.configure', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.start', task ':x-pack:qa:rolling-upgrade-basic:copyRestSpec', task ':x-pack:qa:rolling-upgrade-basic:checkstyle', task ':x-pack:qa:rolling-upgrade-basic:filepermissions', task ':x-pack:qa:rolling-upgrade-basic:forbiddenPatterns', task ':x-pack:qa:rolling-upgrade-basic:jarHell', task ':x-pack:qa:rolling-upgrade-basic:licenseHeaders', task ':x-pack:qa:rolling-upgrade-basic:loggerUsageCheck', task ':x-pack:qa:rolling-upgrade-basic:namingConventions', task ':x-pack:qa:rolling-upgrade-basic:thirdPartyAudit', task ':x-pack:qa:rolling-upgrade-basic:precommit', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#init', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#wait', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestRunner', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.stop', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#mixedClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#mixedClusterTestCluster#clean', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#mixedClusterTestCluster#createCwd', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#mixedClusterTestCluster#checkPrevious', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#mixedClusterTestCluster#stopPrevious', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#mixedClusterTestCluster#extract', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#mixedClusterTestCluster#configure', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#mixedClusterTestCluster#start', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#mixedClusterTestCluster#init', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#mixedClusterTestCluster#wait', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#mixedClusterTestRunner', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.stop', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#init', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#clean', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#createCwd', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#checkPrevious', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#stopPrevious', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#extract', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#configure', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#start', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#wait', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#upgradedClusterTestRunner', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#stop', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#mixedClusterTestCluster#stop', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#upgradedClusterTest', task ':x-pack:qa:rolling-upgrade-basic:v6.4.0-SNAPSHOT#bwcTest', task ':x-pack:qa:rolling-upgrade-basic:integTest', task ':x-pack:qa:rolling-upgrade-basic:test', task ':x-pack:qa:rolling-upgrade-basic:check', task ':x-pack:qa:rolling-upgrade-basic:build', task ':x-pack:qa:saml-idp-tests:copyCheckstyleConf', task ':x-pack:qa:saml-idp-tests:compileTestJava', task ':x-pack:qa:saml-idp-tests:processTestResources', task ':x-pack:qa:saml-idp-tests:testClasses', task ':x-pack:qa:saml-idp-tests:checkstyleTest', task ':x-pack:qa:saml-idp-tests:forbiddenApisTest', task ':x-pack:qa:saml-idp-tests:forbiddenApis', task ':x-pack:qa:saml-idp-tests:copyRestSpec', task ':x-pack:qa:saml-idp-tests:checkstyle', task ':x-pack:qa:saml-idp-tests:filepermissions', task ':x-pack:qa:saml-idp-tests:forbiddenPatterns', task ':x-pack:qa:saml-idp-tests:jarHell', task ':x-pack:qa:saml-idp-tests:licenseHeaders', task ':x-pack:qa:saml-idp-tests:loggerUsageCheck', task ':x-pack:qa:saml-idp-tests:namingConventions', task ':x-pack:qa:saml-idp-tests:thirdPartyAudit', task ':x-pack:qa:saml-idp-tests:precommit', task ':x-pack:qa:saml-idp-tests:integTestCluster#init', task ':x-pack:qa:saml-idp-tests:integTestRunner', task ':x-pack:qa:saml-idp-tests:integTest', task ':x-pack:qa:saml-idp-tests:check', task ':x-pack:qa:saml-idp-tests:build', task ':x-pack:qa:security-client-tests:copyCheckstyleConf', task ':x-pack:qa:security-client-tests:copyXPackPluginProps', task ':x-pack:qa:security-client-tests:compileTestJava', task ':x-pack:qa:security-client-tests:processTestResources', task ':x-pack:qa:security-client-tests:testClasses', task ':x-pack:qa:security-client-tests:checkstyleTest', task ':x-pack:qa:security-client-tests:forbiddenApisTest', task ':x-pack:qa:security-client-tests:forbiddenApis', task ':x-pack:qa:security-client-tests:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:security-client-tests:integTestCluster#clean', task ':x-pack:qa:security-client-tests:integTestCluster#createCwd', task ':x-pack:qa:security-client-tests:integTestCluster#checkPrevious', task ':x-pack:qa:security-client-tests:integTestCluster#stopPrevious', task ':x-pack:qa:security-client-tests:integTestCluster#extract', task ':x-pack:qa:security-client-tests:integTestCluster#configure', task ':x-pack:qa:security-client-tests:integTestCluster#setupDummyUser', task ':x-pack:qa:security-client-tests:integTestCluster#setupTransportClientUser', task ':x-pack:qa:security-client-tests:integTestCluster#start', task ':x-pack:qa:security-client-tests:copyRestSpec', task ':x-pack:qa:security-client-tests:checkstyle', task ':x-pack:qa:security-client-tests:filepermissions', task ':x-pack:qa:security-client-tests:forbiddenPatterns', task ':x-pack:qa:security-client-tests:jarHell', task ':x-pack:qa:security-client-tests:licenseHeaders', task ':x-pack:qa:security-client-tests:loggerUsageCheck', task ':x-pack:qa:security-client-tests:namingConventions', task ':x-pack:qa:security-client-tests:thirdPartyAudit', task ':x-pack:qa:security-client-tests:precommit', task ':x-pack:qa:security-client-tests:integTestCluster#init', task ':x-pack:qa:security-client-tests:integTestCluster#wait', task ':x-pack:qa:security-client-tests:integTestRunner', task ':x-pack:qa:security-client-tests:integTestCluster#stop', task ':x-pack:qa:security-client-tests:integTest', task ':x-pack:qa:security-client-tests:check', task ':x-pack:qa:security-client-tests:build', task ':x-pack:qa:security-example-spi-extension:compileJava', task ':x-pack:qa:security-example-spi-extension:processResources', task ':x-pack:qa:security-example-spi-extension:classes', task ':x-pack:qa:security-example-spi-extension:copyCheckstyleConf', task ':x-pack:qa:security-example-spi-extension:checkstyleMain', task ':x-pack:qa:security-example-spi-extension:copyPluginPropertiesTemplate', task ':x-pack:qa:security-example-spi-extension:pluginProperties', task ':x-pack:qa:security-example-spi-extension:compileTestJava', task ':x-pack:qa:security-example-spi-extension:processTestResources', task ':x-pack:qa:security-example-spi-extension:testClasses', task ':x-pack:qa:security-example-spi-extension:checkstyleTest', task ':x-pack:qa:security-example-spi-extension:forbiddenApisMain', task ':x-pack:qa:security-example-spi-extension:forbiddenApisTest', task ':x-pack:qa:security-example-spi-extension:forbiddenApis', task ':x-pack:qa:security-example-spi-extension:generateNotice', task ':x-pack:qa:security-example-spi-extension:jar', task ':x-pack:qa:security-example-spi-extension:bundlePlugin', task ':x-pack:qa:security-example-spi-extension:buildZip', task ':x-pack:qa:security-example-spi-extension:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:security-example-spi-extension:integTestCluster#clean', task ':x-pack:qa:security-example-spi-extension:integTestCluster#createCwd', task ':x-pack:qa:security-example-spi-extension:integTestCluster#checkPrevious', task ':x-pack:qa:security-example-spi-extension:integTestCluster#stopPrevious', task ':x-pack:qa:security-example-spi-extension:integTestCluster#extract', task ':x-pack:qa:security-example-spi-extension:integTestCluster#configure', task ':x-pack:qa:security-example-spi-extension:integTestCluster#copyPlugins', task ':x-pack:qa:security-example-spi-extension:integTestCluster#installSecurityExampleSpiExtensionPlugin', task ':x-pack:qa:security-example-spi-extension:integTestCluster#setupDummyUser', task ':x-pack:qa:security-example-spi-extension:integTestCluster#start', task ':x-pack:qa:security-example-spi-extension:copyRestSpec', task ':x-pack:qa:security-example-spi-extension:checkstyle', task ':x-pack:qa:security-example-spi-extension:dependencyLicenses', task ':x-pack:qa:security-example-spi-extension:filepermissions', task ':x-pack:qa:security-example-spi-extension:forbiddenPatterns', task ':x-pack:qa:security-example-spi-extension:jarHell', task ':x-pack:qa:security-example-spi-extension:licenseHeaders', task ':x-pack:qa:security-example-spi-extension:loggerUsageCheck', task ':x-pack:qa:security-example-spi-extension:namingConventions', task ':x-pack:qa:security-example-spi-extension:thirdPartyAudit', task ':x-pack:qa:security-example-spi-extension:precommit', task ':x-pack:qa:security-example-spi-extension:test', task ':x-pack:qa:security-example-spi-extension:integTestCluster#init', task ':x-pack:qa:security-example-spi-extension:integTestCluster#wait', task ':x-pack:qa:security-example-spi-extension:integTestRunner', task ':x-pack:qa:security-example-spi-extension:integTestCluster#stop', task ':x-pack:qa:security-example-spi-extension:integTest', task ':x-pack:qa:security-example-spi-extension:extractJar', task ':x-pack:qa:security-example-spi-extension:verifyJarLicense', task ':x-pack:qa:security-example-spi-extension:verifyJarNotice', task ':x-pack:qa:security-example-spi-extension:javadoc', task ':x-pack:qa:security-example-spi-extension:javadocJar', task ':x-pack:qa:security-example-spi-extension:extractJavadocJar', task ':x-pack:qa:security-example-spi-extension:verifyJavadocJarLicense', task ':x-pack:qa:security-example-spi-extension:verifyJavadocJarNotice', task ':x-pack:qa:security-example-spi-extension:sourcesJar', task ':x-pack:qa:security-example-spi-extension:extractSourcesJar', task ':x-pack:qa:security-example-spi-extension:verifySourcesJarLicense', task ':x-pack:qa:security-example-spi-extension:verifySourcesJarNotice', task ':x-pack:qa:security-example-spi-extension:check', task ':x-pack:qa:security-example-spi-extension:build', task ':x-pack:qa:security-migrate-tests:copyCheckstyleConf', task ':x-pack:qa:security-migrate-tests:compileTestJava', task ':x-pack:qa:security-migrate-tests:processTestResources', task ':x-pack:qa:security-migrate-tests:testClasses', task ':x-pack:qa:security-migrate-tests:checkstyleTest', task ':x-pack:qa:security-migrate-tests:forbiddenApisTest', task ':x-pack:qa:security-migrate-tests:forbiddenApis', task ':x-pack:qa:security-migrate-tests:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:security-migrate-tests:integTestCluster#clean', task ':x-pack:qa:security-migrate-tests:integTestCluster#createCwd', task ':x-pack:qa:security-migrate-tests:integTestCluster#checkPrevious', task ':x-pack:qa:security-migrate-tests:integTestCluster#stopPrevious', task ':x-pack:qa:security-migrate-tests:integTestCluster#extract', task ':x-pack:qa:security-migrate-tests:integTestCluster#configure', task ':x-pack:qa:security-migrate-tests:integTestCluster#extraConfig', task ':x-pack:qa:security-migrate-tests:integTestCluster#setupUser#test_admin', task ':x-pack:qa:security-migrate-tests:integTestCluster#setupUser#transport_user', task ':x-pack:qa:security-migrate-tests:integTestCluster#setupUser#existing', task ':x-pack:qa:security-migrate-tests:integTestCluster#setupUser#bob', task ':x-pack:qa:security-migrate-tests:integTestCluster#start', task ':x-pack:qa:security-migrate-tests:copyRestSpec', task ':x-pack:qa:security-migrate-tests:checkstyle', task ':x-pack:qa:security-migrate-tests:filepermissions', task ':x-pack:qa:security-migrate-tests:forbiddenPatterns', task ':x-pack:qa:security-migrate-tests:jarHell', task ':x-pack:qa:security-migrate-tests:licenseHeaders', task ':x-pack:qa:security-migrate-tests:loggerUsageCheck', task ':x-pack:qa:security-migrate-tests:namingConventions', task ':x-pack:qa:security-migrate-tests:thirdPartyAudit', task ':x-pack:qa:security-migrate-tests:precommit', task ':x-pack:qa:security-migrate-tests:integTestCluster#init', task ':x-pack:qa:security-migrate-tests:integTestCluster#wait', task ':x-pack:qa:security-migrate-tests:integTestRunner', task ':x-pack:qa:security-migrate-tests:integTestCluster#stop', task ':x-pack:qa:security-migrate-tests:integTest', task ':x-pack:qa:security-migrate-tests:check', task ':x-pack:qa:security-migrate-tests:build', task ':x-pack:qa:security-setup-password-tests:copyCheckstyleConf', task ':x-pack:qa:security-setup-password-tests:compileTestJava', task ':x-pack:qa:security-setup-password-tests:processTestResources', task ':x-pack:qa:security-setup-password-tests:testClasses', task ':x-pack:qa:security-setup-password-tests:checkstyleTest', task ':x-pack:qa:security-setup-password-tests:forbiddenApisTest', task ':x-pack:qa:security-setup-password-tests:forbiddenApis', task ':x-pack:qa:security-setup-password-tests:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:security-setup-password-tests:integTestCluster#clean', task ':x-pack:qa:security-setup-password-tests:integTestCluster#createCwd', task ':x-pack:qa:security-setup-password-tests:integTestCluster#checkPrevious', task ':x-pack:qa:security-setup-password-tests:integTestCluster#stopPrevious', task ':x-pack:qa:security-setup-password-tests:integTestCluster#extract', task ':x-pack:qa:security-setup-password-tests:integTestCluster#configure', task ':x-pack:qa:security-setup-password-tests:integTestCluster#setupTestAdmin', task ':x-pack:qa:security-setup-password-tests:integTestCluster#start', task ':x-pack:qa:security-setup-password-tests:copyRestSpec', task ':x-pack:qa:security-setup-password-tests:checkstyle', task ':x-pack:qa:security-setup-password-tests:filepermissions', task ':x-pack:qa:security-setup-password-tests:forbiddenPatterns', task ':x-pack:qa:security-setup-password-tests:jarHell', task ':x-pack:qa:security-setup-password-tests:licenseHeaders', task ':x-pack:qa:security-setup-password-tests:loggerUsageCheck', task ':x-pack:qa:security-setup-password-tests:namingConventions', task ':x-pack:qa:security-setup-password-tests:thirdPartyAudit', task ':x-pack:qa:security-setup-password-tests:precommit', task ':x-pack:qa:security-setup-password-tests:integTestCluster#init', task ':x-pack:qa:security-setup-password-tests:integTestCluster#wait', task ':x-pack:qa:security-setup-password-tests:integTestRunner', task ':x-pack:qa:security-setup-password-tests:integTestCluster#stop', task ':x-pack:qa:security-setup-password-tests:integTest', task ':x-pack:qa:security-setup-password-tests:check', task ':x-pack:qa:security-setup-password-tests:build', task ':x-pack:qa:security-tools-tests:copyCheckstyleConf', task ':x-pack:qa:security-tools-tests:compileTestJava', task ':x-pack:qa:security-tools-tests:processTestResources', task ':x-pack:qa:security-tools-tests:testClasses', task ':x-pack:qa:security-tools-tests:checkstyleTest', task ':x-pack:qa:security-tools-tests:forbiddenApisTest', task ':x-pack:qa:security-tools-tests:forbiddenApis', task ':x-pack:qa:security-tools-tests:checkstyle', task ':x-pack:qa:security-tools-tests:filepermissions', task ':x-pack:qa:security-tools-tests:forbiddenPatterns', task ':x-pack:qa:security-tools-tests:jarHell', task ':x-pack:qa:security-tools-tests:licenseHeaders', task ':x-pack:qa:security-tools-tests:loggerUsageCheck', task ':x-pack:qa:security-tools-tests:namingConventions', task ':x-pack:qa:security-tools-tests:thirdPartyAudit', task ':x-pack:qa:security-tools-tests:precommit', task ':x-pack:qa:security-tools-tests:test', task ':x-pack:qa:security-tools-tests:check', task ':x-pack:qa:security-tools-tests:build', task ':x-pack:qa:smoke-test-graph-with-security:copyCheckstyleConf', task ':x-pack:qa:smoke-test-graph-with-security:compileTestJava', task ':x-pack:qa:smoke-test-graph-with-security:processTestResources', task ':x-pack:qa:smoke-test-graph-with-security:testClasses', task ':x-pack:qa:smoke-test-graph-with-security:checkstyleTest', task ':x-pack:qa:smoke-test-graph-with-security:forbiddenApisTest', task ':x-pack:qa:smoke-test-graph-with-security:forbiddenApis', task ':x-pack:qa:smoke-test-graph-with-security:copyGraphRestTests', task ':x-pack:qa:smoke-test-graph-with-security:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:smoke-test-graph-with-security:integTestCluster#clean', task ':x-pack:qa:smoke-test-graph-with-security:integTestCluster#createCwd', task ':x-pack:qa:smoke-test-graph-with-security:integTestCluster#checkPrevious', task ':x-pack:qa:smoke-test-graph-with-security:integTestCluster#stopPrevious', task ':x-pack:qa:smoke-test-graph-with-security:integTestCluster#extract', task ':x-pack:qa:smoke-test-graph-with-security:integTestCluster#configure', task ':x-pack:qa:smoke-test-graph-with-security:integTestCluster#extraConfig', task ':x-pack:qa:smoke-test-graph-with-security:integTestCluster#setupTestAdminUser', task ':x-pack:qa:smoke-test-graph-with-security:integTestCluster#setupGraphExplorerUser', task ':x-pack:qa:smoke-test-graph-with-security:integTestCluster#setupPowerlessUser', task ':x-pack:qa:smoke-test-graph-with-security:integTestCluster#start', task ':x-pack:qa:smoke-test-graph-with-security:copyRestSpec', task ':x-pack:qa:smoke-test-graph-with-security:checkstyle', task ':x-pack:qa:smoke-test-graph-with-security:filepermissions', task ':x-pack:qa:smoke-test-graph-with-security:forbiddenPatterns', task ':x-pack:qa:smoke-test-graph-with-security:jarHell', task ':x-pack:qa:smoke-test-graph-with-security:licenseHeaders', task ':x-pack:qa:smoke-test-graph-with-security:loggerUsageCheck', task ':x-pack:qa:smoke-test-graph-with-security:namingConventions', task ':x-pack:qa:smoke-test-graph-with-security:thirdPartyAudit', task ':x-pack:qa:smoke-test-graph-with-security:precommit', task ':x-pack:qa:smoke-test-graph-with-security:integTestCluster#init', task ':x-pack:qa:smoke-test-graph-with-security:integTestCluster#wait', task ':x-pack:qa:smoke-test-graph-with-security:integTestRunner', task ':x-pack:qa:smoke-test-graph-with-security:integTestCluster#stop', task ':x-pack:qa:smoke-test-graph-with-security:integTest', task ':x-pack:qa:smoke-test-graph-with-security:check', task ':x-pack:qa:smoke-test-graph-with-security:build', task ':x-pack:qa:smoke-test-ml-with-security:copyCheckstyleConf', task ':x-pack:qa:smoke-test-ml-with-security:compileTestJava', task ':x-pack:qa:smoke-test-ml-with-security:processTestResources', task ':x-pack:qa:smoke-test-ml-with-security:testClasses', task ':x-pack:qa:smoke-test-ml-with-security:checkstyleTest', task ':x-pack:qa:smoke-test-ml-with-security:forbiddenApisTest', task ':x-pack:qa:smoke-test-ml-with-security:forbiddenApis', task ':x-pack:qa:smoke-test-ml-with-security:copyMlRestTests', task ':x-pack:qa:smoke-test-ml-with-security:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:smoke-test-ml-with-security:integTestCluster#clean', task ':x-pack:qa:smoke-test-ml-with-security:integTestCluster#createCwd', task ':x-pack:qa:smoke-test-ml-with-security:integTestCluster#checkPrevious', task ':x-pack:qa:smoke-test-ml-with-security:integTestCluster#stopPrevious', task ':x-pack:qa:smoke-test-ml-with-security:integTestCluster#extract', task ':x-pack:qa:smoke-test-ml-with-security:integTestCluster#configure', task ':x-pack:qa:smoke-test-ml-with-security:integTestCluster#extraConfig', task ':x-pack:qa:smoke-test-ml-with-security:integTestCluster#setupTestAdminUser', task ':x-pack:qa:smoke-test-ml-with-security:integTestCluster#setupMlAdminUser', task ':x-pack:qa:smoke-test-ml-with-security:integTestCluster#setupMlUserUser', task ':x-pack:qa:smoke-test-ml-with-security:integTestCluster#setupPowerlessUser', task ':x-pack:qa:smoke-test-ml-with-security:integTestCluster#start', task ':x-pack:qa:smoke-test-ml-with-security:copyRestSpec', task ':x-pack:qa:smoke-test-ml-with-security:checkstyle', task ':x-pack:qa:smoke-test-ml-with-security:filepermissions', task ':x-pack:qa:smoke-test-ml-with-security:forbiddenPatterns', task ':x-pack:qa:smoke-test-ml-with-security:jarHell', task ':x-pack:qa:smoke-test-ml-with-security:licenseHeaders', task ':x-pack:qa:smoke-test-ml-with-security:loggerUsageCheck', task ':x-pack:qa:smoke-test-ml-with-security:namingConventions', task ':x-pack:qa:smoke-test-ml-with-security:thirdPartyAudit', task ':x-pack:qa:smoke-test-ml-with-security:precommit', task ':x-pack:qa:smoke-test-ml-with-security:integTestCluster#init', task ':x-pack:qa:smoke-test-ml-with-security:integTestCluster#wait', task ':x-pack:qa:smoke-test-ml-with-security:integTestRunner', task ':x-pack:qa:smoke-test-ml-with-security:integTestCluster#stop', task ':x-pack:qa:smoke-test-ml-with-security:integTest', task ':x-pack:qa:smoke-test-ml-with-security:check', task ':x-pack:qa:smoke-test-ml-with-security:build', task ':x-pack:qa:smoke-test-monitoring-with-watcher:copyCheckstyleConf', task ':x-pack:qa:smoke-test-monitoring-with-watcher:compileTestJava', task ':x-pack:qa:smoke-test-monitoring-with-watcher:processTestResources', task ':x-pack:qa:smoke-test-monitoring-with-watcher:testClasses', task ':x-pack:qa:smoke-test-monitoring-with-watcher:checkstyleTest', task ':x-pack:qa:smoke-test-monitoring-with-watcher:forbiddenApisTest', task ':x-pack:qa:smoke-test-monitoring-with-watcher:forbiddenApis', task ':x-pack:qa:smoke-test-monitoring-with-watcher:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:smoke-test-monitoring-with-watcher:integTestCluster#clean', task ':x-pack:qa:smoke-test-monitoring-with-watcher:integTestCluster#createCwd', task ':x-pack:qa:smoke-test-monitoring-with-watcher:integTestCluster#checkPrevious', task ':x-pack:qa:smoke-test-monitoring-with-watcher:integTestCluster#stopPrevious', task ':x-pack:qa:smoke-test-monitoring-with-watcher:integTestCluster#extract', task ':x-pack:qa:smoke-test-monitoring-with-watcher:integTestCluster#configure', task ':x-pack:qa:smoke-test-monitoring-with-watcher:integTestCluster#start', task ':x-pack:qa:smoke-test-monitoring-with-watcher:copyRestSpec', task ':x-pack:qa:smoke-test-monitoring-with-watcher:checkstyle', task ':x-pack:qa:smoke-test-monitoring-with-watcher:filepermissions', task ':x-pack:qa:smoke-test-monitoring-with-watcher:forbiddenPatterns', task ':x-pack:qa:smoke-test-monitoring-with-watcher:jarHell', task ':x-pack:qa:smoke-test-monitoring-with-watcher:licenseHeaders', task ':x-pack:qa:smoke-test-monitoring-with-watcher:loggerUsageCheck', task ':x-pack:qa:smoke-test-monitoring-with-watcher:namingConventions', task ':x-pack:qa:smoke-test-monitoring-with-watcher:thirdPartyAudit', task ':x-pack:qa:smoke-test-monitoring-with-watcher:precommit', task ':x-pack:qa:smoke-test-monitoring-with-watcher:integTestCluster#init', task ':x-pack:qa:smoke-test-monitoring-with-watcher:integTestCluster#wait', task ':x-pack:qa:smoke-test-monitoring-with-watcher:integTestRunner', task ':x-pack:qa:smoke-test-monitoring-with-watcher:integTestCluster#stop', task ':x-pack:qa:smoke-test-monitoring-with-watcher:integTest', task ':x-pack:qa:smoke-test-monitoring-with-watcher:check', task ':x-pack:qa:smoke-test-monitoring-with-watcher:build', task ':x-pack:qa:smoke-test-plugins:copyCheckstyleConf', task ':x-pack:qa:smoke-test-plugins:compileTestJava', task ':x-pack:qa:smoke-test-plugins:processTestResources', task ':x-pack:qa:smoke-test-plugins:testClasses', task ':x-pack:qa:smoke-test-plugins:checkstyleTest', task ':x-pack:qa:smoke-test-plugins:forbiddenApisTest', task ':x-pack:qa:smoke-test-plugins:forbiddenApis', task ':x-pack:qa:smoke-test-plugins:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:smoke-test-plugins:integTestCluster#clean', task ':x-pack:qa:smoke-test-plugins:integTestCluster#createCwd', task ':x-pack:qa:smoke-test-plugins:integTestCluster#checkPrevious', task ':x-pack:qa:smoke-test-plugins:integTestCluster#stopPrevious', task ':x-pack:qa:smoke-test-plugins:integTestCluster#extract', task ':x-pack:qa:smoke-test-plugins:integTestCluster#configure', task ':x-pack:qa:smoke-test-plugins:integTestCluster#copyPlugins', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installAnalysisIcuPlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installAnalysisKuromojiPlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installAnalysisNoriPlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installAnalysisPhoneticPlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installAnalysisSmartcnPlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installAnalysisStempelPlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installAnalysisUkrainianPlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installDiscoveryAzureClassicPlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installDiscoveryEc2Plugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installDiscoveryFilePlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installDiscoveryGcePlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installIngestAttachmentPlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installIngestGeoipPlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installIngestUserAgentPlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installMapperMurmur3Plugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installMapperSizePlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installRepositoryAzurePlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installRepositoryGcsPlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installRepositoryHdfsPlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installRepositoryS3Plugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installStoreSmbPlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#installTransportNioPlugin', task ':x-pack:qa:smoke-test-plugins:integTestCluster#setupDummyUser', task ':x-pack:qa:smoke-test-plugins:integTestCluster#start', task ':x-pack:qa:smoke-test-plugins:copyRestSpec', task ':x-pack:qa:smoke-test-plugins:checkstyle', task ':x-pack:qa:smoke-test-plugins:filepermissions', task ':x-pack:qa:smoke-test-plugins:forbiddenPatterns', task ':x-pack:qa:smoke-test-plugins:jarHell', task ':x-pack:qa:smoke-test-plugins:licenseHeaders', task ':x-pack:qa:smoke-test-plugins:loggerUsageCheck', task ':x-pack:qa:smoke-test-plugins:namingConventions', task ':x-pack:qa:smoke-test-plugins:thirdPartyAudit', task ':x-pack:qa:smoke-test-plugins:precommit', task ':x-pack:qa:smoke-test-plugins:integTestCluster#init', task ':x-pack:qa:smoke-test-plugins:integTestCluster#wait', task ':x-pack:qa:smoke-test-plugins:integTestRunner', task ':x-pack:qa:smoke-test-plugins:integTestCluster#stop', task ':x-pack:qa:smoke-test-plugins:integTest', task ':x-pack:qa:smoke-test-plugins:check', task ':x-pack:qa:smoke-test-plugins:build', task ':x-pack:qa:smoke-test-plugins-ssl:copyCheckstyleConf', task ':x-pack:qa:smoke-test-plugins-ssl:copyXPackPluginProps', task ':x-pack:qa:smoke-test-plugins-ssl:compileTestJava', task ':x-pack:qa:smoke-test-plugins-ssl:createClientKeyStore', task ':x-pack:qa:smoke-test-plugins-ssl:createNodeKeyStore', task ':x-pack:qa:smoke-test-plugins-ssl:exportClientCertificate', task ':x-pack:qa:smoke-test-plugins-ssl:importClientCertificateInNodeKeyStore', task ':x-pack:qa:smoke-test-plugins-ssl:exportNodeCertificate', task ':x-pack:qa:smoke-test-plugins-ssl:importNodeCertificateInClientKeyStore', task ':x-pack:qa:smoke-test-plugins-ssl:processTestResources', task ':x-pack:qa:smoke-test-plugins-ssl:testClasses', task ':x-pack:qa:smoke-test-plugins-ssl:checkstyleTest', task ':x-pack:qa:smoke-test-plugins-ssl:forbiddenApisTest', task ':x-pack:qa:smoke-test-plugins-ssl:forbiddenApis', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#clean', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#createCwd', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#checkPrevious', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#stopPrevious', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#extract', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#configure', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#createKeystore', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#addToKeystore#xpack.security.http.ssl.keystore.secure_password', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#copyPlugins', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installAnalysisIcuPlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installAnalysisKuromojiPlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installAnalysisNoriPlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installAnalysisPhoneticPlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installAnalysisSmartcnPlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installAnalysisStempelPlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installAnalysisUkrainianPlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installDiscoveryAzureClassicPlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installDiscoveryEc2Plugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installDiscoveryFilePlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installDiscoveryGcePlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installIngestAttachmentPlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installIngestGeoipPlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installIngestUserAgentPlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installMapperMurmur3Plugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installMapperSizePlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installRepositoryAzurePlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installRepositoryGcsPlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installRepositoryHdfsPlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installRepositoryS3Plugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installStoreSmbPlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#installTransportNioPlugin', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#extraConfig', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#setupTestUser', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#setupMonitoringUser', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#start', task ':x-pack:qa:smoke-test-plugins-ssl:copyRestSpec', task ':x-pack:qa:smoke-test-plugins-ssl:checkstyle', task ':x-pack:qa:smoke-test-plugins-ssl:filepermissions', task ':x-pack:qa:smoke-test-plugins-ssl:forbiddenPatterns', task ':x-pack:qa:smoke-test-plugins-ssl:jarHell', task ':x-pack:qa:smoke-test-plugins-ssl:licenseHeaders', task ':x-pack:qa:smoke-test-plugins-ssl:loggerUsageCheck', task ':x-pack:qa:smoke-test-plugins-ssl:namingConventions', task ':x-pack:qa:smoke-test-plugins-ssl:thirdPartyAudit', task ':x-pack:qa:smoke-test-plugins-ssl:precommit', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#init', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#wait', task ':x-pack:qa:smoke-test-plugins-ssl:integTestRunner', task ':x-pack:qa:smoke-test-plugins-ssl:integTestCluster#stop', task ':x-pack:qa:smoke-test-plugins-ssl:integTest', task ':x-pack:qa:smoke-test-plugins-ssl:check', task ':x-pack:qa:smoke-test-plugins-ssl:build', task ':x-pack:qa:smoke-test-security-with-mustache:copyCheckstyleConf', task ':x-pack:qa:smoke-test-security-with-mustache:compileTestJava', task ':x-pack:qa:smoke-test-security-with-mustache:processTestResources', task ':x-pack:qa:smoke-test-security-with-mustache:testClasses', task ':x-pack:qa:smoke-test-security-with-mustache:checkstyleTest', task ':x-pack:qa:smoke-test-security-with-mustache:forbiddenApisTest', task ':x-pack:qa:smoke-test-security-with-mustache:forbiddenApis', task ':x-pack:qa:smoke-test-security-with-mustache:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:smoke-test-security-with-mustache:integTestCluster#clean', task ':x-pack:qa:smoke-test-security-with-mustache:integTestCluster#createCwd', task ':x-pack:qa:smoke-test-security-with-mustache:integTestCluster#checkPrevious', task ':x-pack:qa:smoke-test-security-with-mustache:integTestCluster#stopPrevious', task ':x-pack:qa:smoke-test-security-with-mustache:integTestCluster#extract', task ':x-pack:qa:smoke-test-security-with-mustache:integTestCluster#configure', task ':x-pack:qa:smoke-test-security-with-mustache:integTestCluster#setupDummyUser', task ':x-pack:qa:smoke-test-security-with-mustache:integTestCluster#start', task ':x-pack:qa:smoke-test-security-with-mustache:copyRestSpec', task ':x-pack:qa:smoke-test-security-with-mustache:checkstyle', task ':x-pack:qa:smoke-test-security-with-mustache:filepermissions', task ':x-pack:qa:smoke-test-security-with-mustache:forbiddenPatterns', task ':x-pack:qa:smoke-test-security-with-mustache:jarHell', task ':x-pack:qa:smoke-test-security-with-mustache:licenseHeaders', task ':x-pack:qa:smoke-test-security-with-mustache:loggerUsageCheck', task ':x-pack:qa:smoke-test-security-with-mustache:namingConventions', task ':x-pack:qa:smoke-test-security-with-mustache:thirdPartyAudit', task ':x-pack:qa:smoke-test-security-with-mustache:precommit', task ':x-pack:qa:smoke-test-security-with-mustache:integTestCluster#init', task ':x-pack:qa:smoke-test-security-with-mustache:integTestCluster#wait', task ':x-pack:qa:smoke-test-security-with-mustache:integTestRunner', task ':x-pack:qa:smoke-test-security-with-mustache:integTestCluster#stop', task ':x-pack:qa:smoke-test-security-with-mustache:integTest', task ':x-pack:qa:smoke-test-security-with-mustache:check', task ':x-pack:qa:smoke-test-security-with-mustache:build', task ':x-pack:qa:smoke-test-watcher:copyCheckstyleConf', task ':x-pack:qa:smoke-test-watcher:compileTestJava', task ':x-pack:qa:smoke-test-watcher:processTestResources', task ':x-pack:qa:smoke-test-watcher:testClasses', task ':x-pack:qa:smoke-test-watcher:checkstyleTest', task ':x-pack:qa:smoke-test-watcher:forbiddenApisTest', task ':x-pack:qa:smoke-test-watcher:forbiddenApis', task ':x-pack:qa:smoke-test-watcher:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:smoke-test-watcher:integTestCluster#clean', task ':x-pack:qa:smoke-test-watcher:integTestCluster#createCwd', task ':x-pack:qa:smoke-test-watcher:integTestCluster#checkPrevious', task ':x-pack:qa:smoke-test-watcher:integTestCluster#stopPrevious', task ':x-pack:qa:smoke-test-watcher:integTestCluster#extract', task ':x-pack:qa:smoke-test-watcher:integTestCluster#configure', task ':x-pack:qa:smoke-test-watcher:integTestCluster#start', task ':x-pack:qa:smoke-test-watcher:copyRestSpec', task ':x-pack:qa:smoke-test-watcher:checkstyle', task ':x-pack:qa:smoke-test-watcher:filepermissions', task ':x-pack:qa:smoke-test-watcher:forbiddenPatterns', task ':x-pack:qa:smoke-test-watcher:jarHell', task ':x-pack:qa:smoke-test-watcher:licenseHeaders', task ':x-pack:qa:smoke-test-watcher:loggerUsageCheck', task ':x-pack:qa:smoke-test-watcher:namingConventions', task ':x-pack:qa:smoke-test-watcher:thirdPartyAudit', task ':x-pack:qa:smoke-test-watcher:precommit', task ':x-pack:qa:smoke-test-watcher:integTestCluster#init', task ':x-pack:qa:smoke-test-watcher:integTestCluster#wait', task ':x-pack:qa:smoke-test-watcher:integTestRunner', task ':x-pack:qa:smoke-test-watcher:integTestCluster#stop', task ':x-pack:qa:smoke-test-watcher:integTest', task ':x-pack:qa:smoke-test-watcher:check', task ':x-pack:qa:smoke-test-watcher:build', task ':x-pack:qa:smoke-test-watcher-with-security:copyCheckstyleConf', task ':x-pack:qa:smoke-test-watcher-with-security:compileTestJava', task ':x-pack:qa:smoke-test-watcher-with-security:processTestResources', task ':x-pack:qa:smoke-test-watcher-with-security:testClasses', task ':x-pack:qa:smoke-test-watcher-with-security:checkstyleTest', task ':x-pack:qa:smoke-test-watcher-with-security:forbiddenApisTest', task ':x-pack:qa:smoke-test-watcher-with-security:forbiddenApis', task ':x-pack:qa:smoke-test-watcher-with-security:copyWatcherRestTests', task ':x-pack:qa:smoke-test-watcher-with-security:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:smoke-test-watcher-with-security:integTestCluster#clean', task ':x-pack:qa:smoke-test-watcher-with-security:integTestCluster#createCwd', task ':x-pack:qa:smoke-test-watcher-with-security:integTestCluster#checkPrevious', task ':x-pack:qa:smoke-test-watcher-with-security:integTestCluster#stopPrevious', task ':x-pack:qa:smoke-test-watcher-with-security:integTestCluster#extract', task ':x-pack:qa:smoke-test-watcher-with-security:integTestCluster#configure', task ':x-pack:qa:smoke-test-watcher-with-security:integTestCluster#extraConfig', task ':x-pack:qa:smoke-test-watcher-with-security:integTestCluster#setupTestAdminUser', task ':x-pack:qa:smoke-test-watcher-with-security:integTestCluster#setupXpackUserForTests', task ':x-pack:qa:smoke-test-watcher-with-security:integTestCluster#setupWatcherManagerUser', task ':x-pack:qa:smoke-test-watcher-with-security:integTestCluster#setupPowerlessUser', task ':x-pack:qa:smoke-test-watcher-with-security:integTestCluster#start', task ':x-pack:qa:smoke-test-watcher-with-security:copyRestSpec', task ':x-pack:qa:smoke-test-watcher-with-security:checkstyle', task ':x-pack:qa:smoke-test-watcher-with-security:filepermissions', task ':x-pack:qa:smoke-test-watcher-with-security:forbiddenPatterns', task ':x-pack:qa:smoke-test-watcher-with-security:jarHell', task ':x-pack:qa:smoke-test-watcher-with-security:licenseHeaders', task ':x-pack:qa:smoke-test-watcher-with-security:loggerUsageCheck', task ':x-pack:qa:smoke-test-watcher-with-security:namingConventions', task ':x-pack:qa:smoke-test-watcher-with-security:thirdPartyAudit', task ':x-pack:qa:smoke-test-watcher-with-security:precommit', task ':x-pack:qa:smoke-test-watcher-with-security:integTestCluster#init', task ':x-pack:qa:smoke-test-watcher-with-security:integTestCluster#wait', task ':x-pack:qa:smoke-test-watcher-with-security:integTestRunner', task ':x-pack:qa:smoke-test-watcher-with-security:integTestCluster#stop', task ':x-pack:qa:smoke-test-watcher-with-security:integTest', task ':x-pack:qa:smoke-test-watcher-with-security:check', task ':x-pack:qa:smoke-test-watcher-with-security:build', task ':x-pack:plugin:sql:jdbc:compileJava', task ':x-pack:plugin:sql:jdbc:processResources', task ':x-pack:plugin:sql:jdbc:classes', task ':x-pack:plugin:sql:jdbc:jar', task ':x-pack:plugin:sql:sql-cli:nodepsJar', task ':x-pack:qa:sql:compileJava', task ':x-pack:qa:sql:processResources', task ':x-pack:qa:sql:classes', task ':x-pack:qa:sql:copyCheckstyleConf', task ':x-pack:qa:sql:checkstyleMain', task ':x-pack:qa:sql:compileTestJava', task ':x-pack:qa:sql:processTestResources', task ':x-pack:qa:sql:testClasses', task ':x-pack:qa:sql:checkstyleTest', task ':x-pack:qa:sql:forbiddenApisMain', task ':x-pack:qa:sql:forbiddenApisTest', task ':x-pack:qa:sql:forbiddenApis', task ':x-pack:qa:sql:checkstyle', task ':x-pack:qa:sql:dependencyLicenses', task ':x-pack:qa:sql:filepermissions', task ':x-pack:qa:sql:forbiddenPatterns', task ':x-pack:qa:sql:jarHell', task ':x-pack:qa:sql:licenseHeaders', task ':x-pack:qa:sql:loggerUsageCheck', task ':x-pack:qa:sql:namingConventions', task ':x-pack:qa:sql:thirdPartyAudit', task ':x-pack:qa:sql:precommit', task ':x-pack:plugin:sql:jdbc:copyCheckstyleConf', task ':x-pack:plugin:sql:jdbc:checkstyleMain', task ':x-pack:plugin:sql:jdbc:compileTestJava', task ':x-pack:plugin:sql:jdbc:processTestResources', task ':x-pack:plugin:sql:jdbc:testClasses', task ':x-pack:plugin:sql:jdbc:checkstyleTest', task ':x-pack:plugin:sql:jdbc:checkstyle', task ':x-pack:plugin:sql:jdbc:dependencyLicenses', task ':x-pack:plugin:sql:jdbc:filepermissions', task ':x-pack:plugin:sql:jdbc:forbiddenApisMain', task ':x-pack:plugin:sql:jdbc:forbiddenApisTest', task ':x-pack:plugin:sql:jdbc:forbiddenApis', task ':x-pack:plugin:sql:jdbc:forbiddenPatterns', task ':x-pack:plugin:sql:jdbc:jarHell', task ':x-pack:plugin:sql:jdbc:licenseHeaders', task ':x-pack:plugin:sql:jdbc:loggerUsageCheck', task ':x-pack:plugin:sql:jdbc:namingConventions', task ':x-pack:plugin:sql:jdbc:thirdPartyAudit', task ':x-pack:plugin:sql:jdbc:precommit', task ':x-pack:plugin:sql:jdbc:test', task ':x-pack:qa:sql:test', task ':x-pack:qa:sql:jar', task ':x-pack:qa:sql:extractJar', task ':x-pack:qa:sql:verifyJarLicense', task ':x-pack:qa:sql:verifyJarNotice', task ':x-pack:plugin:sql:jdbc:javadoc', task ':x-pack:plugin:sql:sql-shared-client:javadoc', task ':x-pack:plugin:sql:sql-cli:javadoc', task ':x-pack:qa:sql:javadoc', task ':x-pack:qa:sql:javadocJar', task ':x-pack:qa:sql:extractJavadocJar', task ':x-pack:qa:sql:verifyJavadocJarLicense', task ':x-pack:qa:sql:verifyJavadocJarNotice', task ':x-pack:qa:sql:sourcesJar', task ':x-pack:qa:sql:extractSourcesJar', task ':x-pack:qa:sql:verifySourcesJarLicense', task ':x-pack:qa:sql:verifySourcesJarNotice', task ':x-pack:qa:sql:check', task ':x-pack:qa:sql:build', task ':x-pack:qa:transport-client-tests:copyCheckstyleConf', task ':x-pack:qa:transport-client-tests:compileTestJava', task ':x-pack:qa:transport-client-tests:processTestResources', task ':x-pack:qa:transport-client-tests:testClasses', task ':x-pack:qa:transport-client-tests:checkstyleTest', task ':x-pack:qa:transport-client-tests:forbiddenApisTest', task ':x-pack:qa:transport-client-tests:forbiddenApis', task ':x-pack:qa:transport-client-tests:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:transport-client-tests:integTestCluster#clean', task ':x-pack:qa:transport-client-tests:integTestCluster#createCwd', task ':x-pack:qa:transport-client-tests:integTestCluster#checkPrevious', task ':x-pack:qa:transport-client-tests:integTestCluster#stopPrevious', task ':x-pack:qa:transport-client-tests:integTestCluster#extract', task ':x-pack:qa:transport-client-tests:integTestCluster#configure', task ':x-pack:qa:transport-client-tests:integTestCluster#start', task ':x-pack:qa:transport-client-tests:copyRestSpec', task ':x-pack:qa:transport-client-tests:checkstyle', task ':x-pack:qa:transport-client-tests:filepermissions', task ':x-pack:qa:transport-client-tests:forbiddenPatterns', task ':x-pack:qa:transport-client-tests:jarHell', task ':x-pack:qa:transport-client-tests:licenseHeaders', task ':x-pack:qa:transport-client-tests:loggerUsageCheck', task ':x-pack:qa:transport-client-tests:namingConventions', task ':x-pack:qa:transport-client-tests:thirdPartyAudit', task ':x-pack:qa:transport-client-tests:precommit', task ':x-pack:qa:transport-client-tests:integTestCluster#init', task ':x-pack:qa:transport-client-tests:integTestCluster#wait', task ':x-pack:qa:transport-client-tests:integTestRunner', task ':x-pack:qa:transport-client-tests:integTestCluster#stop', task ':x-pack:qa:transport-client-tests:integTest', task ':x-pack:qa:transport-client-tests:check', task ':x-pack:qa:transport-client-tests:build', task ':x-pack:test:idp-fixture:compileJava', task ':x-pack:test:idp-fixture:processResources', task ':x-pack:test:idp-fixture:classes', task ':x-pack:test:idp-fixture:jar', task ':x-pack:test:idp-fixture:javadoc', task ':x-pack:test:idp-fixture:javadocJar', task ':x-pack:test:idp-fixture:sourcesJar', task ':x-pack:test:idp-fixture:assemble', task ':x-pack:test:idp-fixture:copyCheckstyleConf', task ':x-pack:test:idp-fixture:checkstyleMain', task ':x-pack:test:idp-fixture:compileTestJava', task ':x-pack:test:idp-fixture:processTestResources', task ':x-pack:test:idp-fixture:testClasses', task ':x-pack:test:idp-fixture:checkstyleTest', task ':x-pack:test:idp-fixture:forbiddenApisMain', task ':x-pack:test:idp-fixture:forbiddenApisTest', task ':x-pack:test:idp-fixture:forbiddenApis', task ':x-pack:test:idp-fixture:checkstyle', task ':x-pack:test:idp-fixture:dependencyLicenses', task ':x-pack:test:idp-fixture:filepermissions', task ':x-pack:test:idp-fixture:forbiddenPatterns', task ':x-pack:test:idp-fixture:jarHell', task ':x-pack:test:idp-fixture:licenseHeaders', task ':x-pack:test:idp-fixture:loggerUsageCheck', task ':x-pack:test:idp-fixture:namingConventions', task ':x-pack:test:idp-fixture:thirdPartyAudit', task ':x-pack:test:idp-fixture:precommit', task ':x-pack:test:idp-fixture:test', task ':x-pack:test:idp-fixture:extractJar', task ':x-pack:test:idp-fixture:verifyJarLicense', task ':x-pack:test:idp-fixture:verifyJarNotice', task ':x-pack:test:idp-fixture:extractJavadocJar', task ':x-pack:test:idp-fixture:verifyJavadocJarLicense', task ':x-pack:test:idp-fixture:verifyJavadocJarNotice', task ':x-pack:test:idp-fixture:extractSourcesJar', task ':x-pack:test:idp-fixture:verifySourcesJarLicense', task ':x-pack:test:idp-fixture:verifySourcesJarNotice', task ':x-pack:test:idp-fixture:check', task ':x-pack:test:idp-fixture:build', task ':x-pack:test:smb-fixture:compileJava', task ':x-pack:test:smb-fixture:processResources', task ':x-pack:test:smb-fixture:classes', task ':x-pack:test:smb-fixture:jar', task ':x-pack:test:smb-fixture:javadoc', task ':x-pack:test:smb-fixture:javadocJar', task ':x-pack:test:smb-fixture:sourcesJar', task ':x-pack:test:smb-fixture:assemble', task ':x-pack:test:smb-fixture:copyCheckstyleConf', task ':x-pack:test:smb-fixture:checkstyleMain', task ':x-pack:test:smb-fixture:compileTestJava', task ':x-pack:test:smb-fixture:processTestResources', task ':x-pack:test:smb-fixture:testClasses', task ':x-pack:test:smb-fixture:checkstyleTest', task ':x-pack:test:smb-fixture:forbiddenApisMain', task ':x-pack:test:smb-fixture:forbiddenApisTest', task ':x-pack:test:smb-fixture:forbiddenApis', task ':x-pack:test:smb-fixture:checkstyle', task ':x-pack:test:smb-fixture:dependencyLicenses', task ':x-pack:test:smb-fixture:filepermissions', task ':x-pack:test:smb-fixture:forbiddenPatterns', task ':x-pack:test:smb-fixture:jarHell', task ':x-pack:test:smb-fixture:licenseHeaders', task ':x-pack:test:smb-fixture:loggerUsageCheck', task ':x-pack:test:smb-fixture:namingConventions', task ':x-pack:test:smb-fixture:thirdPartyAudit', task ':x-pack:test:smb-fixture:precommit', task ':x-pack:test:smb-fixture:test', task ':x-pack:test:smb-fixture:extractJar', task ':x-pack:test:smb-fixture:verifyJarLicense', task ':x-pack:test:smb-fixture:verifyJarNotice', task ':x-pack:test:smb-fixture:extractJavadocJar', task ':x-pack:test:smb-fixture:verifyJavadocJarLicense', task ':x-pack:test:smb-fixture:verifyJavadocJarNotice', task ':x-pack:test:smb-fixture:extractSourcesJar', task ':x-pack:test:smb-fixture:verifySourcesJarLicense', task ':x-pack:test:smb-fixture:verifySourcesJarNotice', task ':x-pack:test:smb-fixture:check', task ':x-pack:test:smb-fixture:build', task ':plugins:repository-azure:qa:microsoft-azure-storage:assemble', task ':plugins:repository-azure:qa:microsoft-azure-storage:build', task ':plugins:repository-gcs:qa:google-cloud-storage:assemble', task ':plugins:repository-gcs:qa:google-cloud-storage:build', task ':plugins:repository-s3:qa:amazon-s3:assemble', task ':plugins:repository-s3:qa:amazon-s3:build', task ':x-pack:plugin:bwc:maintenance-bugfix-snapshot:check', task ':x-pack:plugin:bwc:maintenance-bugfix-snapshot:build', task ':x-pack:plugin:bwc:next-bugfix-snapshot:check', task ':x-pack:plugin:bwc:next-bugfix-snapshot:build', task ':x-pack:plugin:bwc:next-minor-snapshot:check', task ':x-pack:plugin:bwc:next-minor-snapshot:build', task ':x-pack:plugin:bwc:staged-minor-snapshot:check', task ':x-pack:plugin:bwc:staged-minor-snapshot:build', task ':x-pack:plugin:ml:cpp-snapshot:distTar', task ':x-pack:plugin:ml:cpp-snapshot:distZip', task ':x-pack:plugin:ml:cpp-snapshot:assemble', task ':x-pack:plugin:ml:cpp-snapshot:check', task ':x-pack:plugin:ml:cpp-snapshot:build', task ':x-pack:plugin:sql:jdbc:generatePomFileForNebulaPublication', task ':x-pack:plugin:sql:jdbc:javadocJar', task ':x-pack:plugin:sql:jdbc:sourcesJar', task ':x-pack:plugin:sql:jdbc:zipWithDependencies', task ':x-pack:plugin:sql:jdbc:assemble', task ':x-pack:plugin:sql:jdbc:extractJar', task ':x-pack:plugin:sql:jdbc:verifyJarLicense', task ':x-pack:plugin:sql:jdbc:verifyJarNotice', task ':x-pack:plugin:sql:jdbc:extractJavadocJar', task ':x-pack:plugin:sql:jdbc:verifyJavadocJarLicense', task ':x-pack:plugin:sql:jdbc:verifyJavadocJarNotice', task ':x-pack:plugin:sql:jdbc:extractSourcesJar', task ':x-pack:plugin:sql:jdbc:verifySourcesJarLicense', task ':x-pack:plugin:sql:jdbc:verifySourcesJarNotice', task ':x-pack:plugin:sql:jdbc:check', task ':x-pack:plugin:sql:jdbc:build', task ':x-pack:plugin:sql:sql-cli:javadocJar', task ':x-pack:plugin:sql:sql-cli:sourcesJar', task ':x-pack:plugin:sql:sql-cli:assemble', task ':x-pack:plugin:sql:sql-cli:extractJar', task ':x-pack:plugin:sql:sql-cli:verifyJarLicense', task ':x-pack:plugin:sql:sql-cli:verifyJarNotice', task ':x-pack:plugin:sql:sql-cli:extractJavadocJar', task ':x-pack:plugin:sql:sql-cli:verifyJavadocJarLicense', task ':x-pack:plugin:sql:sql-cli:verifyJavadocJarNotice', task ':x-pack:plugin:sql:sql-cli:extractNodepsJar', task ':x-pack:plugin:sql:sql-cli:verifyNodepsJarLicense', task ':x-pack:plugin:sql:sql-cli:verifyNodepsJarNotice', task ':x-pack:plugin:sql:sql-cli:extractSourcesJar', task ':x-pack:plugin:sql:sql-cli:verifySourcesJarLicense', task ':x-pack:plugin:sql:sql-cli:verifySourcesJarNotice', task ':x-pack:plugin:sql:sql-cli:check', task ':x-pack:plugin:sql:sql-cli:build', task ':x-pack:plugin:sql:sql-proto:javadocJar', task ':x-pack:plugin:sql:sql-proto:sourcesJar', task ':x-pack:plugin:sql:sql-proto:assemble', task ':x-pack:plugin:sql:sql-proto:extractJar', task ':x-pack:plugin:sql:sql-proto:verifyJarLicense', task ':x-pack:plugin:sql:sql-proto:verifyJarNotice', task ':x-pack:plugin:sql:sql-proto:extractJavadocJar', task ':x-pack:plugin:sql:sql-proto:verifyJavadocJarLicense', task ':x-pack:plugin:sql:sql-proto:verifyJavadocJarNotice', task ':x-pack:plugin:sql:sql-proto:extractSourcesJar', task ':x-pack:plugin:sql:sql-proto:verifySourcesJarLicense', task ':x-pack:plugin:sql:sql-proto:verifySourcesJarNotice', task ':x-pack:plugin:sql:sql-proto:check', task ':x-pack:plugin:sql:sql-proto:build', task ':x-pack:plugin:sql:sql-shared-client:javadocJar', task ':x-pack:plugin:sql:sql-shared-client:sourcesJar', task ':x-pack:plugin:sql:sql-shared-client:assemble', task ':x-pack:plugin:sql:sql-shared-client:extractJar', task ':x-pack:plugin:sql:sql-shared-client:verifyJarLicense', task ':x-pack:plugin:sql:sql-shared-client:verifyJarNotice', task ':x-pack:plugin:sql:sql-shared-client:extractJavadocJar', task ':x-pack:plugin:sql:sql-shared-client:verifyJavadocJarLicense', task ':x-pack:plugin:sql:sql-shared-client:verifyJavadocJarNotice', task ':x-pack:plugin:sql:sql-shared-client:extractSourcesJar', task ':x-pack:plugin:sql:sql-shared-client:verifySourcesJarLicense', task ':x-pack:plugin:sql:sql-shared-client:verifySourcesJarNotice', task ':x-pack:plugin:sql:sql-shared-client:check', task ':x-pack:plugin:sql:sql-shared-client:build', task ':x-pack:qa:full-cluster-restart:with-system-key:copyCheckstyleConf', task ':x-pack:qa:full-cluster-restart:with-system-key:processTestResources', task ':x-pack:qa:full-cluster-restart:with-system-key:copyXPackRestSpec', task ':x-pack:qa:full-cluster-restart:with-system-key:copyXPackPluginProps', task ':x-pack:qa:full-cluster-restart:with-system-key:compileTestJava', task ':x-pack:qa:full-cluster-restart:with-system-key:testClasses', task ':x-pack:qa:full-cluster-restart:with-system-key:checkstyleTest', task ':x-pack:qa:full-cluster-restart:with-system-key:forbiddenApisTest', task ':x-pack:qa:full-cluster-restart:with-system-key:forbiddenApis', task ':x-pack:qa:full-cluster-restart:with-system-key:copyTestNodeKeystore', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.clean', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.createCwd', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.checkPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.stopPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.extract', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.configure', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.createKeystore', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.addToKeystore#xpack.watcher.encryption_key', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.copyBwcPlugins', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.installPluginPlugin', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.extraConfig', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.setupTestUser', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.start', task ':x-pack:qa:full-cluster-restart:with-system-key:copyRestSpec', task ':x-pack:qa:full-cluster-restart:with-system-key:checkstyle', task ':x-pack:qa:full-cluster-restart:with-system-key:filepermissions', task ':x-pack:qa:full-cluster-restart:with-system-key:forbiddenPatterns', task ':x-pack:qa:full-cluster-restart:with-system-key:jarHell', task ':x-pack:qa:full-cluster-restart:with-system-key:licenseHeaders', task ':x-pack:qa:full-cluster-restart:with-system-key:loggerUsageCheck', task ':x-pack:qa:full-cluster-restart:with-system-key:namingConventions', task ':x-pack:qa:full-cluster-restart:with-system-key:thirdPartyAudit', task ':x-pack:qa:full-cluster-restart:with-system-key:precommit', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#init', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.clean', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.createCwd', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.checkPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.stopPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.extract', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.configure', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.createKeystore', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.addToKeystore#xpack.watcher.encryption_key', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.copyBwcPlugins', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.installPluginPlugin', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.extraConfig', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.setupTestUser', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.start', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#wait', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestRunner', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.stop', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.stop', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.clean', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.createCwd', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.checkPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.stopPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.extract', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.configure', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.createKeystore', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.addToKeystore#xpack.watcher.encryption_key', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.extraConfig', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.setupTestUser', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.start', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#init', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.clean', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.createCwd', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.checkPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.stopPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.extract', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.configure', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.createKeystore', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.addToKeystore#xpack.watcher.encryption_key', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.extraConfig', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.setupTestUser', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.start', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#wait', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestRunner', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.stop', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.stop', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTest', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.2.5-SNAPSHOT#bwcTest', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.clean', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.createCwd', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.checkPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.stopPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.extract', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.configure', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.createKeystore', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.addToKeystore#xpack.watcher.encryption_key', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.extraConfig', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.setupTestUser', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.start', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#init', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.clean', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.createCwd', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.checkPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.stopPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.extract', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.configure', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.createKeystore', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.addToKeystore#xpack.watcher.encryption_key', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.extraConfig', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.setupTestUser', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.start', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#wait', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestRunner', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.stop', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.stop', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.clean', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.createCwd', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.checkPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.stopPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.extract', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.configure', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.createKeystore', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.addToKeystore#xpack.watcher.encryption_key', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.extraConfig', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.setupTestUser', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.start', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#init', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.clean', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.createCwd', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.checkPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.stopPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.extract', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.configure', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.createKeystore', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.addToKeystore#xpack.watcher.encryption_key', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.extraConfig', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.setupTestUser', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.start', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#wait', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestRunner', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.stop', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.stop', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#upgradedClusterTest', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.3.0-SNAPSHOT#bwcTest', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.clean', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.createCwd', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.checkPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.stopPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.extract', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.configure', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.createKeystore', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.addToKeystore#xpack.watcher.encryption_key', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.extraConfig', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.setupTestUser', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.start', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#init', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.clean', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.createCwd', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.checkPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.stopPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.extract', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.configure', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.createKeystore', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.addToKeystore#xpack.watcher.encryption_key', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.extraConfig', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.setupTestUser', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.start', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#wait', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestRunner', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.stop', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.stop', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.clean', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.createCwd', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.checkPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.stopPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.extract', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.configure', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.createKeystore', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.addToKeystore#xpack.watcher.encryption_key', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.extraConfig', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.setupTestUser', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.start', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#init', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.clean', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.createCwd', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.checkPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.stopPrevious', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.extract', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.configure', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.createKeystore', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.addToKeystore#xpack.watcher.encryption_key', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.extraConfig', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.setupTestUser', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.start', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#wait', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestRunner', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.stop', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.stop', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTest', task ':x-pack:qa:full-cluster-restart:with-system-key:v6.4.0-SNAPSHOT#bwcTest', task ':x-pack:qa:full-cluster-restart:with-system-key:integTest', task ':x-pack:qa:full-cluster-restart:with-system-key:test', task ':x-pack:qa:full-cluster-restart:with-system-key:check', task ':x-pack:qa:full-cluster-restart:with-system-key:build', task ':x-pack:qa:full-cluster-restart:without-system-key:copyCheckstyleConf', task ':x-pack:qa:full-cluster-restart:without-system-key:processTestResources', task ':x-pack:qa:full-cluster-restart:without-system-key:copyXPackRestSpec', task ':x-pack:qa:full-cluster-restart:without-system-key:copyXPackPluginProps', task ':x-pack:qa:full-cluster-restart:without-system-key:compileTestJava', task ':x-pack:qa:full-cluster-restart:without-system-key:testClasses', task ':x-pack:qa:full-cluster-restart:without-system-key:checkstyleTest', task ':x-pack:qa:full-cluster-restart:without-system-key:forbiddenApisTest', task ':x-pack:qa:full-cluster-restart:without-system-key:forbiddenApis', task ':x-pack:qa:full-cluster-restart:without-system-key:copyTestNodeKeystore', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.clean', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.createCwd', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.checkPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.stopPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.extract', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.configure', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.copyBwcPlugins', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.installPluginPlugin', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.extraConfig', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.setupTestUser', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.start', task ':x-pack:qa:full-cluster-restart:without-system-key:copyRestSpec', task ':x-pack:qa:full-cluster-restart:without-system-key:checkstyle', task ':x-pack:qa:full-cluster-restart:without-system-key:filepermissions', task ':x-pack:qa:full-cluster-restart:without-system-key:forbiddenPatterns', task ':x-pack:qa:full-cluster-restart:without-system-key:jarHell', task ':x-pack:qa:full-cluster-restart:without-system-key:licenseHeaders', task ':x-pack:qa:full-cluster-restart:without-system-key:loggerUsageCheck', task ':x-pack:qa:full-cluster-restart:without-system-key:namingConventions', task ':x-pack:qa:full-cluster-restart:without-system-key:thirdPartyAudit', task ':x-pack:qa:full-cluster-restart:without-system-key:precommit', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#init', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.clean', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.createCwd', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.checkPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.stopPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.extract', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.configure', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.copyBwcPlugins', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.installPluginPlugin', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.extraConfig', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.setupTestUser', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.start', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#wait', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestRunner', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node1.stop', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#oldClusterTestCluster#node0.stop', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.clean', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.createCwd', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.checkPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.stopPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.extract', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.configure', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.createKeystore', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.extraConfig', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.setupTestUser', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.start', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#init', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.clean', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.createCwd', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.checkPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.stopPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.extract', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.configure', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.createKeystore', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.extraConfig', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.setupTestUser', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.start', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#wait', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestRunner', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node1.stop', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestCluster#node0.stop', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#upgradedClusterTest', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.2.5-SNAPSHOT#bwcTest', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.clean', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.createCwd', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.checkPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.stopPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.extract', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.configure', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.extraConfig', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.setupTestUser', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.start', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#init', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.clean', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.createCwd', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.checkPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.stopPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.extract', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.configure', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.extraConfig', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.setupTestUser', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.start', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#wait', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestRunner', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node1.stop', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#oldClusterTestCluster#node0.stop', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.clean', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.createCwd', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.checkPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.stopPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.extract', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.configure', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.createKeystore', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.extraConfig', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.setupTestUser', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.start', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#init', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.clean', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.createCwd', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.checkPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.stopPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.extract', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.configure', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.createKeystore', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.extraConfig', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.setupTestUser', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.start', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#wait', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestRunner', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node1.stop', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTestCluster#node0.stop', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#upgradedClusterTest', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.3.0-SNAPSHOT#bwcTest', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.clean', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.createCwd', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.checkPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.stopPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.extract', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.configure', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.extraConfig', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.setupTestUser', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.start', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#init', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.clean', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.createCwd', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.checkPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.stopPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.extract', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.configure', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.extraConfig', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.setupTestUser', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.start', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#wait', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestRunner', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.stop', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.stop', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.clean', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.createCwd', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.checkPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.stopPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.extract', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.configure', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.createKeystore', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.extraConfig', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.setupTestUser', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.start', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#init', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.clean', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.createCwd', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.checkPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.stopPrevious', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.extract', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.configure', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.createKeystore', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.extraConfig', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.setupTestUser', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.start', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#wait', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestRunner', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node1.stop', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#node0.stop', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTest', task ':x-pack:qa:full-cluster-restart:without-system-key:v6.4.0-SNAPSHOT#bwcTest', task ':x-pack:qa:full-cluster-restart:without-system-key:integTest', task ':x-pack:qa:full-cluster-restart:without-system-key:test', task ':x-pack:qa:full-cluster-restart:without-system-key:check', task ':x-pack:qa:full-cluster-restart:without-system-key:build', task ':x-pack:qa:rolling-upgrade:with-system-key:copyCheckstyleConf', task ':x-pack:qa:rolling-upgrade:with-system-key:processTestResources', task ':x-pack:qa:rolling-upgrade:with-system-key:copyXPackRestSpec', task ':x-pack:qa:rolling-upgrade:with-system-key:copyXPackPluginProps', task ':x-pack:qa:rolling-upgrade:with-system-key:compileTestJava', task ':x-pack:qa:rolling-upgrade:with-system-key:testClasses', task ':x-pack:qa:rolling-upgrade:with-system-key:checkstyleTest', task ':x-pack:qa:rolling-upgrade:with-system-key:forbiddenApisTest', task ':x-pack:qa:rolling-upgrade:with-system-key:forbiddenApis', task ':x-pack:qa:rolling-upgrade:with-system-key:copyTestNodeKeystore', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.clean', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.createCwd', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.checkPrevious', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.stopPrevious', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.extract', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.configure', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.createKeystore', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.addToKeystore#xpack.watcher.encryption_key', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.extraConfig', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.setupTestUser', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.start', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.clean', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.createCwd', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.checkPrevious', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.stopPrevious', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.extract', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.configure', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.createKeystore', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.addToKeystore#xpack.watcher.encryption_key', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.extraConfig', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.setupTestUser', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.start', task ':x-pack:qa:rolling-upgrade:with-system-key:copyRestSpec', task ':x-pack:qa:rolling-upgrade:with-system-key:checkstyle', task ':x-pack:qa:rolling-upgrade:with-system-key:filepermissions', task ':x-pack:qa:rolling-upgrade:with-system-key:forbiddenPatterns', task ':x-pack:qa:rolling-upgrade:with-system-key:jarHell', task ':x-pack:qa:rolling-upgrade:with-system-key:licenseHeaders', task ':x-pack:qa:rolling-upgrade:with-system-key:loggerUsageCheck', task ':x-pack:qa:rolling-upgrade:with-system-key:namingConventions', task ':x-pack:qa:rolling-upgrade:with-system-key:thirdPartyAudit', task ':x-pack:qa:rolling-upgrade:with-system-key:precommit', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#init', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#wait', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestRunner', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.stop', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#clean', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#createCwd', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#checkPrevious', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#stopPrevious', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#extract', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#configure', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#createKeystore', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#addToKeystore#xpack.watcher.encryption_key', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#extraConfig', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#setupTestUser', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#start', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#init', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#wait', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestRunner', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.stop', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#init', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#clean', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#createCwd', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#checkPrevious', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#stopPrevious', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#extract', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#configure', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#createKeystore', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#addToKeystore#xpack.watcher.encryption_key', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#extraConfig', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#setupTestUser', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#start', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#wait', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestRunner', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#stop', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#stop', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTest', task ':x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#bwcTest', task ':x-pack:qa:rolling-upgrade:with-system-key:integTest', task ':x-pack:qa:rolling-upgrade:with-system-key:test', task ':x-pack:qa:rolling-upgrade:with-system-key:check', task ':x-pack:qa:rolling-upgrade:with-system-key:build', task ':x-pack:qa:rolling-upgrade:without-system-key:copyCheckstyleConf', task ':x-pack:qa:rolling-upgrade:without-system-key:processTestResources', task ':x-pack:qa:rolling-upgrade:without-system-key:copyXPackRestSpec', task ':x-pack:qa:rolling-upgrade:without-system-key:copyXPackPluginProps', task ':x-pack:qa:rolling-upgrade:without-system-key:compileTestJava', task ':x-pack:qa:rolling-upgrade:without-system-key:testClasses', task ':x-pack:qa:rolling-upgrade:without-system-key:checkstyleTest', task ':x-pack:qa:rolling-upgrade:without-system-key:forbiddenApisTest', task ':x-pack:qa:rolling-upgrade:without-system-key:forbiddenApis', task ':x-pack:qa:rolling-upgrade:without-system-key:copyTestNodeKeystore', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.clean', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.createCwd', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.checkPrevious', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.stopPrevious', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.extract', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.configure', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.extraConfig', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.setupTestUser', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.start', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.clean', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.createCwd', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.checkPrevious', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.stopPrevious', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.extract', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.configure', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.extraConfig', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.setupTestUser', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.start', task ':x-pack:qa:rolling-upgrade:without-system-key:copyRestSpec', task ':x-pack:qa:rolling-upgrade:without-system-key:checkstyle', task ':x-pack:qa:rolling-upgrade:without-system-key:filepermissions', task ':x-pack:qa:rolling-upgrade:without-system-key:forbiddenPatterns', task ':x-pack:qa:rolling-upgrade:without-system-key:jarHell', task ':x-pack:qa:rolling-upgrade:without-system-key:licenseHeaders', task ':x-pack:qa:rolling-upgrade:without-system-key:loggerUsageCheck', task ':x-pack:qa:rolling-upgrade:without-system-key:namingConventions', task ':x-pack:qa:rolling-upgrade:without-system-key:thirdPartyAudit', task ':x-pack:qa:rolling-upgrade:without-system-key:precommit', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#init', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#wait', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestRunner', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node1.stop', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#clean', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#createCwd', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#checkPrevious', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#stopPrevious', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#extract', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#configure', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#createKeystore', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#extraConfig', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#setupTestUser', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#start', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#init', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#wait', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#mixedClusterTestRunner', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#oldClusterTestCluster#node0.stop', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#init', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#clean', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#createCwd', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#checkPrevious', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#stopPrevious', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#extract', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#configure', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#createKeystore', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#extraConfig', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#setupTestUser', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#start', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#wait', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestRunner', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestCluster#stop', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#mixedClusterTestCluster#stop', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#upgradedClusterTest', task ':x-pack:qa:rolling-upgrade:without-system-key:v6.4.0-SNAPSHOT#bwcTest', task ':x-pack:qa:rolling-upgrade:without-system-key:integTest', task ':x-pack:qa:rolling-upgrade:without-system-key:test', task ':x-pack:qa:rolling-upgrade:without-system-key:check', task ':x-pack:qa:rolling-upgrade:without-system-key:build', task ':x-pack:qa:sql:multinode:copyCheckstyleConf', task ':x-pack:qa:sql:multinode:compileTestJava', task ':x-pack:qa:sql:multinode:processTestResources', task ':x-pack:qa:sql:multinode:testClasses', task ':x-pack:qa:sql:multinode:checkstyleTest', task ':x-pack:qa:sql:multinode:forbiddenApisTest', task ':x-pack:qa:sql:multinode:forbiddenApis', task ':x-pack:qa:sql:multinode:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:sql:multinode:integTestCluster#node0.clean', task ':x-pack:qa:sql:multinode:integTestCluster#node0.createCwd', task ':x-pack:qa:sql:multinode:integTestCluster#node0.checkPrevious', task ':x-pack:qa:sql:multinode:integTestCluster#node0.stopPrevious', task ':x-pack:qa:sql:multinode:integTestCluster#node0.extract', task ':x-pack:qa:sql:multinode:integTestCluster#node0.configure', task ':x-pack:qa:sql:multinode:integTestCluster#node0.start', task ':x-pack:qa:sql:multinode:copyRestSpec', task ':x-pack:qa:sql:multinode:checkstyle', task ':x-pack:qa:sql:multinode:filepermissions', task ':x-pack:qa:sql:multinode:forbiddenPatterns', task ':x-pack:qa:sql:multinode:jarHell', task ':x-pack:qa:sql:multinode:licenseHeaders', task ':x-pack:qa:sql:multinode:loggerUsageCheck', task ':x-pack:qa:sql:multinode:namingConventions', task ':x-pack:qa:sql:multinode:thirdPartyAudit', task ':x-pack:qa:sql:multinode:precommit', task ':x-pack:qa:sql:multinode:integTestCluster#init', task ':x-pack:qa:sql:multinode:integTestCluster#node1.clean', task ':x-pack:qa:sql:multinode:integTestCluster#node1.createCwd', task ':x-pack:qa:sql:multinode:integTestCluster#node1.checkPrevious', task ':x-pack:qa:sql:multinode:integTestCluster#node1.stopPrevious', task ':x-pack:qa:sql:multinode:integTestCluster#node1.extract', task ':x-pack:qa:sql:multinode:integTestCluster#node1.configure', task ':x-pack:qa:sql:multinode:integTestCluster#node1.start', task ':x-pack:qa:sql:multinode:integTestCluster#wait', task ':x-pack:qa:sql:multinode:integTestRunner', task ':x-pack:qa:sql:multinode:integTestCluster#node1.stop', task ':x-pack:qa:sql:multinode:integTestCluster#node0.stop', task ':x-pack:qa:sql:multinode:integTest', task ':x-pack:qa:sql:multinode:check', task ':x-pack:qa:sql:multinode:build', task ':x-pack:qa:sql:no-security:copyCheckstyleConf', task ':x-pack:qa:sql:no-security:compileTestJava', task ':x-pack:qa:sql:no-security:processTestResources', task ':x-pack:qa:sql:no-security:testClasses', task ':x-pack:qa:sql:no-security:checkstyleTest', task ':x-pack:qa:sql:no-security:forbiddenApisTest', task ':x-pack:qa:sql:no-security:forbiddenApis', task ':x-pack:qa:sql:no-security:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:sql:no-security:integTestCluster#clean', task ':x-pack:qa:sql:no-security:integTestCluster#createCwd', task ':x-pack:qa:sql:no-security:integTestCluster#checkPrevious', task ':x-pack:qa:sql:no-security:integTestCluster#stopPrevious', task ':x-pack:qa:sql:no-security:integTestCluster#extract', task ':x-pack:qa:sql:no-security:integTestCluster#configure', task ':x-pack:qa:sql:no-security:integTestCluster#start', task ':x-pack:qa:sql:no-security:copyRestSpec', task ':x-pack:qa:sql:no-security:checkstyle', task ':x-pack:qa:sql:no-security:filepermissions', task ':x-pack:qa:sql:no-security:forbiddenPatterns', task ':x-pack:qa:sql:no-security:jarHell', task ':x-pack:qa:sql:no-security:licenseHeaders', task ':x-pack:qa:sql:no-security:loggerUsageCheck', task ':x-pack:qa:sql:no-security:namingConventions', task ':x-pack:qa:sql:no-security:thirdPartyAudit', task ':x-pack:qa:sql:no-security:precommit', task ':x-pack:qa:sql:no-security:integTestCluster#init', task ':x-pack:qa:sql:no-security:integTestCluster#wait', task ':x-pack:qa:sql:no-security:integTestRunner', task ':x-pack:qa:sql:no-security:integTestCluster#stop', task ':x-pack:qa:sql:no-security:integTest', task ':x-pack:qa:sql:no-security:check', task ':x-pack:qa:sql:no-security:build', task ':x-pack:qa:sql:security:copyCheckstyleConf', task ':x-pack:qa:sql:security:compileTestJava', task ':x-pack:qa:sql:security:processTestResources', task ':x-pack:qa:sql:security:testClasses', task ':x-pack:qa:sql:security:checkstyleTest', task ':x-pack:qa:sql:security:forbiddenApisTest', task ':x-pack:qa:sql:security:forbiddenApis', task ':x-pack:qa:sql:security:checkstyle', task ':x-pack:qa:sql:security:filepermissions', task ':x-pack:qa:sql:security:forbiddenPatterns', task ':x-pack:qa:sql:security:jarHell', task ':x-pack:qa:sql:security:licenseHeaders', task ':x-pack:qa:sql:security:loggerUsageCheck', task ':x-pack:qa:sql:security:namingConventions', task ':x-pack:qa:sql:security:thirdPartyAudit', task ':x-pack:qa:sql:security:precommit', task ':x-pack:qa:sql:security:check', task ':x-pack:qa:sql:security:build', task ':x-pack:qa:third-party:active-directory:copyCheckstyleConf', task ':x-pack:qa:third-party:active-directory:compileTestJava', task ':x-pack:qa:third-party:active-directory:processTestResources', task ':x-pack:qa:third-party:active-directory:testClasses', task ':x-pack:qa:third-party:active-directory:checkstyleTest', task ':x-pack:qa:third-party:active-directory:forbiddenApisTest', task ':x-pack:qa:third-party:active-directory:forbiddenApis', task ':x-pack:qa:third-party:active-directory:checkstyle', task ':x-pack:qa:third-party:active-directory:filepermissions', task ':x-pack:qa:third-party:active-directory:forbiddenPatterns', task ':x-pack:qa:third-party:active-directory:jarHell', task ':x-pack:qa:third-party:active-directory:licenseHeaders', task ':x-pack:qa:third-party:active-directory:loggerUsageCheck', task ':x-pack:qa:third-party:active-directory:namingConventions', task ':x-pack:qa:third-party:active-directory:thirdPartyAudit', task ':x-pack:qa:third-party:active-directory:precommit', task ':x-pack:qa:third-party:active-directory:test', task ':x-pack:qa:third-party:active-directory:check', task ':x-pack:qa:third-party:active-directory:build', task ':x-pack:qa:third-party:hipchat:copyCheckstyleConf', task ':x-pack:qa:third-party:hipchat:compileTestJava', task ':x-pack:qa:third-party:hipchat:processTestResources', task ':x-pack:qa:third-party:hipchat:testClasses', task ':x-pack:qa:third-party:hipchat:checkstyleTest', task ':x-pack:qa:third-party:hipchat:forbiddenApisTest', task ':x-pack:qa:third-party:hipchat:forbiddenApis', task ':x-pack:qa:third-party:hipchat:copyRestSpec', task ':x-pack:qa:third-party:hipchat:checkstyle', task ':x-pack:qa:third-party:hipchat:filepermissions', task ':x-pack:qa:third-party:hipchat:forbiddenPatterns', task ':x-pack:qa:third-party:hipchat:jarHell', task ':x-pack:qa:third-party:hipchat:licenseHeaders', task ':x-pack:qa:third-party:hipchat:loggerUsageCheck', task ':x-pack:qa:third-party:hipchat:namingConventions', task ':x-pack:qa:third-party:hipchat:thirdPartyAudit', task ':x-pack:qa:third-party:hipchat:precommit', task ':x-pack:qa:third-party:hipchat:integTestCluster#init', task ':x-pack:qa:third-party:hipchat:integTestRunner', task ':x-pack:qa:third-party:hipchat:integTest', task ':x-pack:qa:third-party:hipchat:check', task ':x-pack:qa:third-party:hipchat:build', task ':x-pack:qa:third-party:jira:copyCheckstyleConf', task ':x-pack:qa:third-party:jira:compileTestJava', task ':x-pack:qa:third-party:jira:processTestResources', task ':x-pack:qa:third-party:jira:testClasses', task ':x-pack:qa:third-party:jira:checkstyleTest', task ':x-pack:qa:third-party:jira:forbiddenApisTest', task ':x-pack:qa:third-party:jira:forbiddenApis', task ':x-pack:qa:third-party:jira:copyRestSpec', task ':x-pack:qa:third-party:jira:checkstyle', task ':x-pack:qa:third-party:jira:filepermissions', task ':x-pack:qa:third-party:jira:forbiddenPatterns', task ':x-pack:qa:third-party:jira:jarHell', task ':x-pack:qa:third-party:jira:licenseHeaders', task ':x-pack:qa:third-party:jira:loggerUsageCheck', task ':x-pack:qa:third-party:jira:namingConventions', task ':x-pack:qa:third-party:jira:thirdPartyAudit', task ':x-pack:qa:third-party:jira:precommit', task ':x-pack:qa:third-party:jira:integTestCluster#init', task ':x-pack:qa:third-party:jira:integTestRunner', task ':x-pack:qa:third-party:jira:integTest', task ':x-pack:qa:third-party:jira:check', task ':x-pack:qa:third-party:jira:build', task ':x-pack:qa:third-party:pagerduty:copyCheckstyleConf', task ':x-pack:qa:third-party:pagerduty:compileTestJava', task ':x-pack:qa:third-party:pagerduty:processTestResources', task ':x-pack:qa:third-party:pagerduty:testClasses', task ':x-pack:qa:third-party:pagerduty:checkstyleTest', task ':x-pack:qa:third-party:pagerduty:forbiddenApisTest', task ':x-pack:qa:third-party:pagerduty:forbiddenApis', task ':x-pack:qa:third-party:pagerduty:copyRestSpec', task ':x-pack:qa:third-party:pagerduty:checkstyle', task ':x-pack:qa:third-party:pagerduty:filepermissions', task ':x-pack:qa:third-party:pagerduty:forbiddenPatterns', task ':x-pack:qa:third-party:pagerduty:jarHell', task ':x-pack:qa:third-party:pagerduty:licenseHeaders', task ':x-pack:qa:third-party:pagerduty:loggerUsageCheck', task ':x-pack:qa:third-party:pagerduty:namingConventions', task ':x-pack:qa:third-party:pagerduty:thirdPartyAudit', task ':x-pack:qa:third-party:pagerduty:precommit', task ':x-pack:qa:third-party:pagerduty:integTestCluster#init', task ':x-pack:qa:third-party:pagerduty:integTestRunner', task ':x-pack:qa:third-party:pagerduty:integTest', task ':x-pack:qa:third-party:pagerduty:check', task ':x-pack:qa:third-party:pagerduty:build', task ':x-pack:qa:third-party:slack:copyCheckstyleConf', task ':x-pack:qa:third-party:slack:compileTestJava', task ':x-pack:qa:third-party:slack:processTestResources', task ':x-pack:qa:third-party:slack:testClasses', task ':x-pack:qa:third-party:slack:checkstyleTest', task ':x-pack:qa:third-party:slack:forbiddenApisTest', task ':x-pack:qa:third-party:slack:forbiddenApis', task ':x-pack:qa:third-party:slack:copyRestSpec', task ':x-pack:qa:third-party:slack:checkstyle', task ':x-pack:qa:third-party:slack:filepermissions', task ':x-pack:qa:third-party:slack:forbiddenPatterns', task ':x-pack:qa:third-party:slack:jarHell', task ':x-pack:qa:third-party:slack:licenseHeaders', task ':x-pack:qa:third-party:slack:loggerUsageCheck', task ':x-pack:qa:third-party:slack:namingConventions', task ':x-pack:qa:third-party:slack:thirdPartyAudit', task ':x-pack:qa:third-party:slack:precommit', task ':x-pack:qa:third-party:slack:integTestCluster#init', task ':x-pack:qa:third-party:slack:integTestRunner', task ':x-pack:qa:third-party:slack:integTest', task ':x-pack:qa:third-party:slack:check', task ':x-pack:qa:third-party:slack:build', task ':x-pack:qa:sql:security:no-ssl:copyCheckstyleConf', task ':x-pack:qa:sql:security:no-ssl:compileTestJava', task ':x-pack:qa:sql:security:no-ssl:processTestResources', task ':x-pack:qa:sql:security:no-ssl:testClasses', task ':x-pack:qa:sql:security:no-ssl:checkstyleTest', task ':x-pack:qa:sql:security:no-ssl:forbiddenApisTest', task ':x-pack:qa:sql:security:no-ssl:forbiddenApis', task ':x-pack:qa:sql:security:no-ssl:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:sql:security:no-ssl:integTestCluster#clean', task ':x-pack:qa:sql:security:no-ssl:integTestCluster#createCwd', task ':x-pack:qa:sql:security:no-ssl:integTestCluster#checkPrevious', task ':x-pack:qa:sql:security:no-ssl:integTestCluster#stopPrevious', task ':x-pack:qa:sql:security:no-ssl:integTestCluster#extract', task ':x-pack:qa:sql:security:no-ssl:integTestCluster#configure', task ':x-pack:qa:sql:security:no-ssl:integTestCluster#extraConfig', task ':x-pack:qa:sql:security:no-ssl:integTestCluster#setupUser#test_admin', task ':x-pack:qa:sql:security:no-ssl:integTestCluster#start', task ':x-pack:qa:sql:security:no-ssl:copyRestSpec', task ':x-pack:qa:sql:security:no-ssl:checkstyle', task ':x-pack:qa:sql:security:no-ssl:filepermissions', task ':x-pack:qa:sql:security:no-ssl:forbiddenPatterns', task ':x-pack:qa:sql:security:no-ssl:jarHell', task ':x-pack:qa:sql:security:no-ssl:licenseHeaders', task ':x-pack:qa:sql:security:no-ssl:loggerUsageCheck', task ':x-pack:qa:sql:security:no-ssl:namingConventions', task ':x-pack:qa:sql:security:no-ssl:thirdPartyAudit', task ':x-pack:qa:sql:security:no-ssl:precommit', task ':x-pack:qa:sql:security:no-ssl:integTestCluster#init', task ':x-pack:qa:sql:security:no-ssl:integTestCluster#wait', task ':x-pack:qa:sql:security:no-ssl:integTestRunner', task ':x-pack:qa:sql:security:no-ssl:integTestCluster#stop', task ':x-pack:qa:sql:security:no-ssl:integTest', task ':x-pack:qa:sql:security:no-ssl:check', task ':x-pack:qa:sql:security:no-ssl:build', task ':x-pack:qa:sql:security:ssl:copyCheckstyleConf', task ':x-pack:qa:sql:security:ssl:compileTestJava', task ':x-pack:qa:sql:security:ssl:createClientKeyStore', task ':x-pack:qa:sql:security:ssl:createNodeKeyStore', task ':x-pack:qa:sql:security:ssl:exportClientCertificate', task ':x-pack:qa:sql:security:ssl:importClientCertificateInNodeKeyStore', task ':x-pack:qa:sql:security:ssl:exportNodeCertificate', task ':x-pack:qa:sql:security:ssl:importNodeCertificateInClientKeyStore', task ':x-pack:qa:sql:security:ssl:processTestResources', task ':x-pack:qa:sql:security:ssl:testClasses', task ':x-pack:qa:sql:security:ssl:checkstyleTest', task ':x-pack:qa:sql:security:ssl:forbiddenApisTest', task ':x-pack:qa:sql:security:ssl:forbiddenApis', task ':x-pack:qa:sql:security:ssl:integTestCluster#prepareCluster.cleanShared', task ':x-pack:qa:sql:security:ssl:integTestCluster#clean', task ':x-pack:qa:sql:security:ssl:integTestCluster#createCwd', task ':x-pack:qa:sql:security:ssl:integTestCluster#checkPrevious', task ':x-pack:qa:sql:security:ssl:integTestCluster#stopPrevious', task ':x-pack:qa:sql:security:ssl:integTestCluster#extract', task ':x-pack:qa:sql:security:ssl:integTestCluster#configure', task ':x-pack:qa:sql:security:ssl:integTestCluster#createKeystore', task ':x-pack:qa:sql:security:ssl:integTestCluster#addToKeystore#xpack.ssl.keystore.secure_password', task ':x-pack:qa:sql:security:ssl:integTestCluster#extraConfig', task ':x-pack:qa:sql:security:ssl:integTestCluster#setupUser#test_admin', task ':x-pack:qa:sql:security:ssl:integTestCluster#start', task ':x-pack:qa:sql:security:ssl:copyRestSpec', task ':x-pack:qa:sql:security:ssl:checkstyle', task ':x-pack:qa:sql:security:ssl:filepermissions', task ':x-pack:qa:sql:security:ssl:forbiddenPatterns', task ':x-pack:qa:sql:security:ssl:jarHell', task ':x-pack:qa:sql:security:ssl:licenseHeaders', task ':x-pack:qa:sql:security:ssl:loggerUsageCheck', task ':x-pack:qa:sql:security:ssl:namingConventions', task ':x-pack:qa:sql:security:ssl:thirdPartyAudit', task ':x-pack:qa:sql:security:ssl:precommit', task ':x-pack:qa:sql:security:ssl:integTestCluster#init', task ':x-pack:qa:sql:security:ssl:integTestCluster#wait', task ':x-pack:qa:sql:security:ssl:integTestRunner', task ':x-pack:qa:sql:security:ssl:integTestCluster#stop', task ':x-pack:qa:sql:security:ssl:integTest', task ':x-pack:qa:sql:security:ssl:check', task ':x-pack:qa:sql:security:ssl:build'] :libs:elasticsearch-core:compileJava (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:compileJava Task ':libs:elasticsearch-core:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':libs:elasticsearch-core:compileJava'. :libs:elasticsearch-core:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.05 secs. Created jar classpath snapshot for incremental compilation in 0.001 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :libs:elasticsearch-core:compileJava (Thread[Daemon worker,5,main]) completed. Took 2.274 secs. :libs:elasticsearch-core:processResources (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\src\main\resources', not found Skipping task ':libs:elasticsearch-core:processResources' as it has no source files and no previous output files. :libs:elasticsearch-core:processResources (Thread[Daemon worker,5,main]) completed. Took 0.001 secs. :libs:elasticsearch-core:classes (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:classes Skipping task ':libs:elasticsearch-core:classes' as it has no actions. :libs:elasticsearch-core:classes (Thread[Daemon worker,5,main]) completed. Took 0.001 secs. :libs:elasticsearch-core:compileJava9Java (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:compileJava9Java Task ':libs:elasticsearch-core:compileJava9Java' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':libs:elasticsearch-core:compileJava9Java'. :libs:elasticsearch-core:compileJava9Java - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\tmp\compileJava9Java\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.006 secs. Created jar classpath snapshot for incremental compilation in 0.0 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :libs:elasticsearch-core:compileJava9Java (Thread[Daemon worker,5,main]) completed. Took 1.314 secs. :libs:elasticsearch-core:processJava9Resources (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:processJava9Resources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\src\java9\resources', not found Skipping task ':libs:elasticsearch-core:processJava9Resources' as it has no source files and no previous output files. :libs:elasticsearch-core:processJava9Resources (Thread[Daemon worker,5,main]) completed. Took 0.003 secs. :libs:elasticsearch-core:java9Classes (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:java9Classes Skipping task ':libs:elasticsearch-core:java9Classes' as it has no actions. :libs:elasticsearch-core:java9Classes (Thread[Daemon worker,5,main]) completed. Took 0.001 secs. :libs:elasticsearch-core:jar (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:jar Task ':libs:elasticsearch-core:jar' is not up-to-date because: No history is available. :libs:elasticsearch-core:jar (Thread[Daemon worker,5,main]) completed. Took 0.169 secs. :libs:plugin-classloader:compileJava (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:compileJava Task ':libs:plugin-classloader:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':libs:plugin-classloader:compileJava'. :libs:plugin-classloader:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.002 secs. Created jar classpath snapshot for incremental compilation in 0.0 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :libs:plugin-classloader:compileJava (Thread[Daemon worker,5,main]) completed. Took 1.064 secs. :libs:plugin-classloader:processResources (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader\src\main\resources', not found Skipping task ':libs:plugin-classloader:processResources' as it has no source files and no previous output files. :libs:plugin-classloader:processResources (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:plugin-classloader:classes (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:classes Skipping task ':libs:plugin-classloader:classes' as it has no actions. :libs:plugin-classloader:classes (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:plugin-classloader:jar (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:jar Task ':libs:plugin-classloader:jar' is not up-to-date because: No history is available. :libs:plugin-classloader:jar (Thread[Daemon worker,5,main]) completed. Took 0.026 secs. :libs:secure-sm:compileJava (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:compileJava Task ':libs:secure-sm:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':libs:secure-sm:compileJava'. :libs:secure-sm:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.004 secs. Created jar classpath snapshot for incremental compilation in 0.0 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :libs:secure-sm:compileJava (Thread[Daemon worker,5,main]) completed. Took 1.186 secs. :libs:secure-sm:processResources (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\src\main\resources', not found Skipping task ':libs:secure-sm:processResources' as it has no source files and no previous output files. :libs:secure-sm:processResources (Thread[Daemon worker,5,main]) completed. Took 0.004 secs. :libs:secure-sm:classes (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:classes Skipping task ':libs:secure-sm:classes' as it has no actions. :libs:secure-sm:classes (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:secure-sm:jar (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:jar Task ':libs:secure-sm:jar' is not up-to-date because: No history is available. :libs:secure-sm:jar (Thread[Daemon worker,5,main]) completed. Took 0.045 secs. :libs:x-content:compileJava (Thread[Daemon worker,5,main]) started. > Task :libs:x-content:compileJava Task ':libs:x-content:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':libs:x-content:compileJava'. :libs:x-content:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\src\main\java\org\elasticsearch\common\xcontent\json\JsonXContentGenerator.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. Class dependency analysis for incremental compilation took 0.075 secs. Created jar classpath snapshot for incremental compilation in 0.277 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :libs:x-content:compileJava (Thread[Daemon worker,5,main]) completed. Took 3.244 secs. :libs:x-content:processResources (Thread[Daemon worker,5,main]) started. > Task :libs:x-content:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\src\main\resources', not found Skipping task ':libs:x-content:processResources' as it has no source files and no previous output files. :libs:x-content:processResources (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:x-content:classes (Thread[Daemon worker,5,main]) started. > Task :libs:x-content:classes Skipping task ':libs:x-content:classes' as it has no actions. :libs:x-content:classes (Thread[Daemon worker,5,main]) completed. Took 0.001 secs. :libs:x-content:jar (Thread[Daemon worker,5,main]) started. > Task :libs:x-content:jar Task ':libs:x-content:jar' is not up-to-date because: No history is available. :libs:x-content:jar (Thread[Daemon worker,5,main]) completed. Took 0.075 secs. :server:cli:compileJava (Thread[Daemon worker,5,main]) started. > Task :server:cli:compileJava Task ':server:cli:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':server:cli:compileJava'. :server:cli:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.008 secs. Created jar classpath snapshot for incremental compilation in 0.034 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :server:cli:compileJava (Thread[Daemon worker,5,main]) completed. Took 1.4 secs. :server:cli:processResources (Thread[Daemon worker,5,main]) started. > Task :server:cli:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\src\main\resources', not found Skipping task ':server:cli:processResources' as it has no source files and no previous output files. :server:cli:processResources (Thread[Daemon worker,5,main]) completed. Took 0.001 secs. :server:cli:classes (Thread[Daemon worker,5,main]) started. > Task :server:cli:classes Skipping task ':server:cli:classes' as it has no actions. :server:cli:classes (Thread[Daemon worker,5,main]) completed. Took 0.001 secs. :server:cli:jar (Thread[Daemon worker,5,main]) started. > Task :server:cli:jar Task ':server:cli:jar' is not up-to-date because: No history is available. :server:cli:jar (Thread[Daemon worker,5,main]) completed. Took 0.025 secs. :server:compileJava (Thread[Daemon worker,5,main]) started. > Task :server:compileJava Task ':server:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':server:compileJava'. :server:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. Class dependency analysis for incremental compilation took 2.492 secs. Created jar classpath snapshot for incremental compilation in 2.39 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :server:compileJava (Thread[Daemon worker,5,main]) completed. Took 42.825 secs. :server:generateModulesList (Thread[Daemon worker,5,main]) started. > Task :server:generateModulesList Task ':server:generateModulesList' is not up-to-date because: No history is available. :server:generateModulesList (Thread[Daemon worker,5,main]) completed. Took 0.015 secs. :server:generatePluginsList (Thread[Daemon worker,5,main]) started. > Task :server:generatePluginsList Task ':server:generatePluginsList' is not up-to-date because: No history is available. :server:generatePluginsList (Thread[Daemon worker,5,main]) completed. Took 0.001 secs. :server:processResources (Thread[Daemon worker,5,main]) started. > Task :server:processResources Task ':server:processResources' is not up-to-date because: No history is available. :server:processResources (Thread[Daemon worker,5,main]) completed. Took 0.032 secs. :server:classes (Thread[Daemon worker,5,main]) started. > Task :server:classes Skipping task ':server:classes' as it has no actions. :server:classes (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :server:compileJava9Java (Thread[Daemon worker,5,main]) started. > Task :server:compileJava9Java Task ':server:compileJava9Java' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':server:compileJava9Java'. :server:compileJava9Java - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\compileJava9Java\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.148 secs. Created jar classpath snapshot for incremental compilation in 0.003 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :server:compileJava9Java (Thread[Daemon worker,5,main]) completed. Took 3.044 secs. :server:processJava9Resources (Thread[Daemon worker,5,main]) started. > Task :server:processJava9Resources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\src\java9\resources', not found Skipping task ':server:processJava9Resources' as it has no source files and no previous output files. :server:processJava9Resources (Thread[Daemon worker,5,main]) completed. Took 0.004 secs. :server:java9Classes (Thread[Daemon worker,5,main]) started. > Task :server:java9Classes Skipping task ':server:java9Classes' as it has no actions. :server:java9Classes (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :server:jar (Thread[Daemon worker,5,main]) started. > Task :server:jar Task ':server:jar' is not up-to-date because: No history is available. :server:jar (Thread[Daemon worker,5,main]) completed. Took 2.118 secs. :benchmarks:compileJava (Thread[Daemon worker,5,main]) started. > Task :benchmarks:compileJava Task ':benchmarks:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':benchmarks:compileJava'. :benchmarks:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.007 secs. Created jar classpath snapshot for incremental compilation in 0.536 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :benchmarks:compileJava (Thread[Daemon worker,5,main]) completed. Took 4.181 secs. :benchmarks:processResources (Thread[Daemon worker,5,main]) started. > Task :benchmarks:processResources Task ':benchmarks:processResources' is not up-to-date because: No history is available. :benchmarks:processResources (Thread[Daemon worker,5,main]) completed. Took 0.003 secs. :benchmarks:classes (Thread[Daemon worker,5,main]) started. > Task :benchmarks:classes Skipping task ':benchmarks:classes' as it has no actions. :benchmarks:classes (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :benchmarks:copyCheckstyleConf (Thread[Daemon worker,5,main]) started. > Task :benchmarks:copyCheckstyleConf Task ':benchmarks:copyCheckstyleConf' is not up-to-date because: No history is available. :benchmarks:copyCheckstyleConf (Thread[Daemon worker,5,main]) completed. Took 0.013 secs. :benchmarks:checkstyleMain (Thread[Daemon worker,5,main]) started. > Task :benchmarks:checkstyleMain Task ':benchmarks:checkstyleMain' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 2 files :benchmarks:checkstyleMain (Thread[Daemon worker,5,main]) completed. Took 1.26 secs. :benchmarks:compileTestJava (Thread[Daemon worker,5,main]) started. > Task :benchmarks:compileTestJava NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\src\test\java', not found Skipping task ':benchmarks:compileTestJava' as it has no source files and no previous output files. :benchmarks:compileTestJava (Thread[Daemon worker,5,main]) completed. Took 0.001 secs. :benchmarks:processTestResources (Thread[Daemon worker,5,main]) started. > Task :benchmarks:processTestResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\src\test\resources', not found Skipping task ':benchmarks:processTestResources' as it has no source files and no previous output files. :benchmarks:processTestResources (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :benchmarks:testClasses (Thread[Daemon worker,5,main]) started. > Task :benchmarks:testClasses UP-TO-DATE Skipping task ':benchmarks:testClasses' as it has no actions. :benchmarks:testClasses (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :benchmarks:checkstyleTest (Thread[Daemon worker,5,main]) started. > Task :benchmarks:checkstyleTest NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\src\test\java', not found Skipping task ':benchmarks:checkstyleTest' as it has no source files and no previous output files. :benchmarks:checkstyleTest (Thread[Daemon worker,5,main]) completed. Took 0.001 secs. :benchmarks:forbiddenApisMain (Thread[Daemon worker,5,main]) started. > Task :benchmarks:forbiddenApisMain Task ':benchmarks:forbiddenApisMain' is not up-to-date because: No history is available. Reading bundled API signatures: jdk-unsafe-1.8 Reading bundled API signatures: jdk-deprecated-1.8 Reading bundled API signatures: jdk-non-portable Reading bundled API signatures: jdk-system-out Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/jdk-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-all-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-server-signatures.txt Loading classes to check... Scanning classes for violations... Forbidden method invocation: java.lang.Class#getDeclaredField(java.lang.String) [Do not violate java's access system: Use getField() instead] in org.elasticsearch.benchmark.routing.allocation.generated.AllocationBenchmark_measureAllocation_jmhTest (AllocationBenchmark_measureAllocation_jmhTest.java:435) Forbidden method invocation: java.lang.reflect.AccessibleObject#setAccessible(boolean) [Do not violate java's access system] in org.elasticsearch.benchmark.routing.allocation.generated.AllocationBenchmark_measureAllocation_jmhTest (AllocationBenchmark_measureAllocation_jmhTest.java:436) Scanned 8 class file(s) for forbidden API invocations (in 0.17s), 2 error(s). :benchmarks:forbiddenApisMain (Thread[Daemon worker,5,main]) completed. Took 0.202 secs. :benchmarks:forbiddenApisTest (Thread[Daemon worker,5,main]) started. > Task :benchmarks:forbiddenApisTest NO-SOURCE Skipping task ':benchmarks:forbiddenApisTest' as it has no source files and no previous output files. :benchmarks:forbiddenApisTest (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :benchmarks:forbiddenApis (Thread[Daemon worker,5,main]) started. > Task :benchmarks:forbiddenApis Skipping task ':benchmarks:forbiddenApis' as it has no actions. :benchmarks:forbiddenApis (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :benchmarks:checkstyle (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :benchmarks:checkstyle Skipping task ':benchmarks:checkstyle' as it has no actions. :benchmarks:checkstyle (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :benchmarks:dependencyLicenses (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :benchmarks:dependencyLicenses SKIPPED Skipping task ':benchmarks:dependencyLicenses' as task onlyIf is false. :benchmarks:dependencyLicenses (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :benchmarks:filepermissions (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :benchmarks:filepermissions SKIPPED Skipping task ':benchmarks:filepermissions' as task onlyIf is false. :benchmarks:filepermissions (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :benchmarks:forbiddenPatterns (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :benchmarks:forbiddenPatterns Task ':benchmarks:forbiddenPatterns' is not up-to-date because: No history is available. file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\src\test\resources', not found file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\src\test\java', not found :benchmarks:forbiddenPatterns (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.027 secs. :benchmarks:jarHell (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :benchmarks:jarHell Task ':benchmarks:jarHell' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks Command: C:\Users\jenkins\.java\java8\bin\java -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\classes\java\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\resources\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.openjdk.jmh\jmh-core\1.19\1ea93b88f8154f0a35c16b46d76cfb2febcf4916\jmh-core-1.19.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.openjdk.jmh\jmh-generator-annprocess\1.19\e5bb13308963df412877e88fede84c1bd869ca03\jmh-generator-annprocess-1.19.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\4.6\306816fb57cf94f108a43c95731b08934dcae15c\jopt-simple-4.6.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.commons\commons-math3\3.2\ec2544ab27e110d2d431bdad7d538ed509b21e62\commons-math3-3.2.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar org.elasticsearch.bootstrap.JarHell Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' checking for jar hell... java.class.path: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\classes\java\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\resources\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.openjdk.jmh\jmh-core\1.19\1ea93b88f8154f0a35c16b46d76cfb2febcf4916\jmh-core-1.19.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.openjdk.jmh\jmh-generator-annprocess\1.19\e5bb13308963df412877e88fede84c1bd869ca03\jmh-generator-annprocess-1.19.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\4.6\306816fb57cf94f108a43c95731b08934dcae15c\jopt-simple-4.6.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.commons\commons-math3\3.2\ec2544ab27e110d2d431bdad7d538ed509b21e62\commons-math3-3.2.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar sun.boot.class.path: C:\Users\jenkins\.java\java8\jre\lib\resources.jar;C:\Users\jenkins\.java\java8\jre\lib\rt.jar;C:\Users\jenkins\.java\java8\jre\lib\sunrsasign.jar;C:\Users\jenkins\.java\java8\jre\lib\jsse.jar;C:\Users\jenkins\.java\java8\jre\lib\jce.jar;C:\Users\jenkins\.java\java8\jre\lib\charsets.jar;C:\Users\jenkins\.java\java8\jre\lib\jfr.jar;C:\Users\jenkins\.java\java8\jre\classes classloader urls: [file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/benchmarks/build/classes/java/main/, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/benchmarks/build/resources/main/, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/server/build/distributions/elasticsearch-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.openjdk.jmh/jmh-core/1.19/1ea93b88f8154f0a35c16b46d76cfb2febcf4916/jmh-core-1.19.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.openjdk.jmh/jmh-generator-annprocess/1.19/e5bb13308963df412877e88fede84c1bd869ca03/jmh-generator-annprocess-1.19.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/4.6/306816fb57cf94f108a43c95731b08934dcae15c/jopt-simple-4.6.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-math3/3.2/ec2544ab27e110d2d431bdad7d538ed509b21e62/commons-math3-3.2.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/x-content/build/distributions/elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/server/cli/build/distributions/elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/elasticsearch-core/build/distributions/elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/secure-sm/build/distributions/elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/7.4.0-snapshot-1ed95c097b/b06e4f6514256a3f187a9892e520638b9c59e63/lucene-core-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/7.4.0-snapshot-1ed95c097b/2b2be48f6622c150496e755497e7bdb8daa46030/lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-backward-codecs/7.4.0-snapshot-1ed95c097b/6cbafc48e8ac4966377665eb3bbe93f9addf04a5/lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-grouping/7.4.0-snapshot-1ed95c097b/4c71cef87fe513a7a96c2a7980ed6f7c2b015763/lucene-grouping-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-highlighter/7.4.0-snapshot-1ed95c097b/665e044d1180100940bccd7e8e41dde48e342da3/lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-join/7.4.0-snapshot-1ed95c097b/d343bbf5792f5969288b59b51179acd29d04f4ee/lucene-join-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-memory/7.4.0-snapshot-1ed95c097b/8915f3c93af3348655bcc204289f9011835738a2/lucene-memory-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-misc/7.4.0-snapshot-1ed95c097b/e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d/lucene-misc-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queries/7.4.0-snapshot-1ed95c097b/5946d5e2be276f66e9ff6d6111acabb03a9330d9/lucene-queries-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/7.4.0-snapshot-1ed95c097b/d9fc5fc63f3d861e5af72e11373368e8a4c6bba6/lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/7.4.0-snapshot-1ed95c097b/ee283c0a1a717f3e0915de75864a93d043efaee3/lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial/7.4.0-snapshot-1ed95c097b/e1adf0220a7c052ac81e2919ffac24ac0e5b007c/lucene-spatial-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial-extras/7.4.0-snapshot-1ed95c097b/6d9306053942c48f43392a634f11a95462b5996e/lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial3d/7.4.0-snapshot-1ed95c097b/2334e8c5f4d0f98659b30e0c2035296e4aae8ff5/lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-suggest/7.4.0-snapshot-1ed95c097b/f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463/lucene-suggest-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.carrotsearch/hppc/0.7.1/8b5057f74ea378c0150a1860874a3ebdcb713767/hppc-0.7.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/joda-time/joda-time/2.9.9/f7b520c458572890807d143670c9b24f4de90897/joda-time-2.9.9.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.tdunning/t-digest/3.2/2ab94758b0276a8a26102adf8d528cf6d0567b9a/t-digest-3.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.hdrhistogram/HdrHistogram/2.1.9/e4631ce165eb400edecfa32e03d3f1be53dee754/HdrHistogram-2.1.9.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.locationtech.spatial4j/spatial4j/0.7/faa8ba85d503da4ab872d17ba8c00da0098ab2f2/spatial4j-0.7.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.locationtech.jts/jts-core/1.15.0/705981b7e25d05a76a3654e597dab6ba423eb79e/jts-core-1.15.0.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.9.1/7a2999229464e7a324aa503c0a52ec0f05efe7bd/log4j-api-2.9.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.9.1/c041978c686866ee8534f538c6220238db3bb6be/log4j-core-2.9.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-1.2-api/2.9.1/894f96d677880d4ab834a1356f62b875e579caaa/log4j-1.2-api-2.9.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.elasticsearch/jna/4.5.1/da10908ae23dc59b19dc258e63aea1c44621dc3a/jna-4.5.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.yaml/snakeyaml/1.17/7a27ea250c5130b2922b86dea63cbb1cc10a660c/snakeyaml-1.17.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.8.10/eb21a035c66ad307e66ec8fce37f5d50fd62d039/jackson-core-2.8.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-smile/2.8.10/e853081fadaad3e98ed801937acc3d8f77580686/jackson-dataformat-smile-2.8.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-yaml/2.8.10/1e08caf1d787c825307d8cc6362452086020d853/jackson-dataformat-yaml-2.8.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-cbor/2.8.10/1c58cc9313ddf19f0900cd61ed044874278ce320/jackson-dataformat-cbor-2.8.10.jar] java.home: C:\Users\jenkins\.java\java8\jre examining directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\classes\java\main examining directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\resources\main examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.openjdk.jmh\jmh-core\1.19\1ea93b88f8154f0a35c16b46d76cfb2febcf4916\jmh-core-1.19.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.openjdk.jmh\jmh-generator-annprocess\1.19\e5bb13308963df412877e88fede84c1bd869ca03\jmh-generator-annprocess-1.19.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\4.6\306816fb57cf94f108a43c95731b08934dcae15c\jopt-simple-4.6.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.commons\commons-math3\3.2\ec2544ab27e110d2d431bdad7d538ed509b21e62\commons-math3-3.2.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar no jar hell found :benchmarks:jarHell (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.477 secs. :benchmarks:licenseHeaders (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :benchmarks:licenseHeaders file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\src\test\java', not found Task ':benchmarks:licenseHeaders' is not up-to-date because: No history is available. :benchmarks:licenseHeaders (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.483 secs. :test:logger-usage:compileJava (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :test:logger-usage:compileJava Task ':test:logger-usage:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':test:logger-usage:compileJava'. :test:logger-usage:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.006 secs. Created jar classpath snapshot for incremental compilation in 0.065 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :test:logger-usage:compileJava (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 1.458 secs. :test:logger-usage:processResources (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :test:logger-usage:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\src\main\resources', not found Skipping task ':test:logger-usage:processResources' as it has no source files and no previous output files. :test:logger-usage:processResources (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :test:logger-usage:classes (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :test:logger-usage:classes Skipping task ':test:logger-usage:classes' as it has no actions. :test:logger-usage:classes (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :test:logger-usage:jar (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :test:logger-usage:jar Task ':test:logger-usage:jar' is not up-to-date because: No history is available. :test:logger-usage:jar (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.022 secs. :benchmarks:loggerUsageCheck (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :benchmarks:loggerUsageCheck Task ':benchmarks:loggerUsageCheck' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks Command: C:\Users\jenkins\.java\java8\bin\java -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\distributions\logger-usage-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.ow2.asm\asm-debug-all\5.0.4\702b8525fcf81454235e5e2fa2a35f15ffc0ec7e\asm-debug-all-5.0.4.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar org.elasticsearch.test.loggerusage.ESLoggerUsageChecker C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\classes\java\main Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' checking for wrong usages of ESLogger... No wrong usages found :benchmarks:loggerUsageCheck (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.302 secs. :build-tools:compileJava (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:compileJava Task ':build-tools:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':build-tools:compileJava'. :build-tools:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.004 secs. Created jar classpath snapshot for incremental compilation in 11.559 secs. 1556 duplicate classes found in classpath (see all with --debug). Written jar classpath snapshot for incremental compilation in 0.0 secs. :build-tools:compileJava (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 13.137 secs. :build-tools:compileGroovy (Thread[Task worker for ':' Thread 2,5,main]) started. This JVM does not support getting OS memory, so no OS memory status updates will be broadcast Initialized native services in: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\native > Task :build-tools:compileGroovy Task ':build-tools:compileGroovy' is not up-to-date because: No history is available. Starting process 'Gradle Worker Daemon 1'. Working directory: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\workers Command: C:\Users\jenkins\.java\java8\bin\java.exe -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager -Dfile.encoding=windows-1252 -Duser.country=US -Duser.language=en -Duser.variant -cp C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\4.7\workerMain\gradle-worker.jar worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Worker Daemon 1' Successfully started process 'Gradle Worker Daemon 1' Started Gradle worker daemon (0.555 secs) with fork options DaemonForkOptions{executable=C:\Users\jenkins\.java\java8\bin\java.exe, minHeapSize=null, maxHeapSize=null, jvmArgs=[], classpath=[C:\Users\jenkins.SLAVE-0633DD87F\.gradle\wrapper\dists\gradle-4.7-all\4cret0dgl5o3b21weaoncl7ys\gradle-4.7\lib\groovy-all-2.4.12.jar, C:\Users\jenkins.SLAVE-0633DD87F\.gradle\wrapper\dists\gradle-4.7-all\4cret0dgl5o3b21weaoncl7ys\gradle-4.7\lib\ant-1.9.9.jar, C:\Users\jenkins.SLAVE-0633DD87F\.gradle\wrapper\dists\gradle-4.7-all\4cret0dgl5o3b21weaoncl7ys\gradle-4.7\lib\ant-launcher-1.9.9.jar], keepAliveMode=SESSION}. Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc Command: C:\Users\jenkins\.java\java10\bin\javac.exe @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\tmp\compileGroovy\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' :build-tools:compileGroovy (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 9.944 secs. :build-tools:writeVersionProperties (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:writeVersionProperties Task ':build-tools:writeVersionProperties' is not up-to-date because: No history is available. :build-tools:writeVersionProperties (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.006 secs. :build-tools:processResources (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:processResources Task ':build-tools:processResources' is not up-to-date because: No history is available. :build-tools:processResources (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.048 secs. :build-tools:classes (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:classes Skipping task ':build-tools:classes' as it has no actions. :build-tools:classes (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :build-tools:jar (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:jar Task ':build-tools:jar' is not up-to-date because: No history is available. :build-tools:jar (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.212 secs. :benchmarks:namingConventions (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :benchmarks:namingConventions SKIPPED Skipping task ':benchmarks:namingConventions' as task onlyIf is false. :benchmarks:namingConventions (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :benchmarks:thirdPartyAudit (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :benchmarks:thirdPartyAudit Task ':benchmarks:thirdPartyAudit' is not up-to-date because: No history is available. [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.openjdk.jmh\jmh-core\1.19\1ea93b88f8154f0a35c16b46d76cfb2febcf4916\jmh-core-1.19.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.openjdk.jmh\jmh-generator-annprocess\1.19\e5bb13308963df412877e88fede84c1bd869ca03\jmh-generator-annprocess-1.19.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\4.6\306816fb57cf94f108a43c95731b08934dcae15c\jopt-simple-4.6.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.commons\commons-math3\3.2\ec2544ab27e110d2d431bdad7d538ed509b21e62\commons-math3-3.2.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\tmp\thirdPartyAudit [thirdPartyAudit] Reading inline API signatures... [thirdPartyAudit] Loading classes to check... [thirdPartyAudit] Scanning classes for violations... [thirdPartyAudit] Forbidden class/interface use: sun.management.HotspotThreadMBean [non-public internal runtime class] [thirdPartyAudit] in org.openjdk.jmh.profile.HotspotThreadProfiler (HotspotThreadProfiler.java:45) [thirdPartyAudit] Forbidden class/interface use: sun.misc.Unsafe [non-public internal runtime class] [thirdPartyAudit] in org.openjdk.jmh.util.Utils (Utils.java, field declaration of 'U') [thirdPartyAudit] Forbidden class/interface use: sun.misc.Unsafe [non-public internal runtime class] [thirdPartyAudit] in org.openjdk.jmh.util.Utils (Utils.java:357) [thirdPartyAudit] Forbidden class/interface use: sun.misc.Unsafe [non-public internal runtime class] [thirdPartyAudit] in org.openjdk.jmh.util.Utils (Utils.java:44) [thirdPartyAudit] Forbidden class/interface use: sun.management.HotspotRuntimeMBean [non-public internal runtime class] [thirdPartyAudit] in org.openjdk.jmh.profile.HotspotRuntimeProfiler (HotspotRuntimeProfiler.java:46) [thirdPartyAudit] Forbidden class/interface use: sun.management.HotspotMemoryMBean [non-public internal runtime class] [thirdPartyAudit] in org.openjdk.jmh.profile.HotspotMemoryProfiler (HotspotMemoryProfiler.java:39) [thirdPartyAudit] Forbidden class/interface use: sun.management.counter.Counter [non-public internal runtime class] [thirdPartyAudit] in org.openjdk.jmh.profile.AbstractHotspotProfiler (AbstractHotspotProfiler.java:67) [thirdPartyAudit] Forbidden class/interface use: sun.management.counter.Counter [non-public internal runtime class] [thirdPartyAudit] in org.openjdk.jmh.profile.AbstractHotspotProfiler (AbstractHotspotProfiler.java:67) [thirdPartyAudit] Forbidden class/interface use: sun.management.counter.Counter [non-public internal runtime class] [thirdPartyAudit] in org.openjdk.jmh.profile.AbstractHotspotProfiler (AbstractHotspotProfiler.java:83) [thirdPartyAudit] Forbidden class/interface use: sun.management.counter.Counter [non-public internal runtime class] [thirdPartyAudit] in org.openjdk.jmh.profile.AbstractHotspotProfiler (AbstractHotspotProfiler.java:85) [thirdPartyAudit] Forbidden class/interface use: sun.management.counter.Counter [non-public internal runtime class] [thirdPartyAudit] in org.openjdk.jmh.profile.AbstractHotspotProfiler (AbstractHotspotProfiler.java:86) [thirdPartyAudit] Forbidden class/interface use: sun.management.counter.Counter [non-public internal runtime class] [thirdPartyAudit] in org.openjdk.jmh.profile.AbstractHotspotProfiler (AbstractHotspotProfiler.java:87) [thirdPartyAudit] Forbidden class/interface use: sun.management.counter.Counter [non-public internal runtime class] [thirdPartyAudit] in org.openjdk.jmh.profile.AbstractHotspotProfiler (AbstractHotspotProfiler.java:87) [thirdPartyAudit] Forbidden class/interface use: sun.management.HotspotClassLoadingMBean [non-public internal runtime class] [thirdPartyAudit] in org.openjdk.jmh.profile.HotspotClassloadingProfiler (HotspotClassloadingProfiler.java:39) [thirdPartyAudit] Forbidden class/interface use: sun.management.HotspotCompilationMBean [non-public internal runtime class] [thirdPartyAudit] in org.openjdk.jmh.profile.HotspotCompilationProfiler (HotspotCompilationProfiler.java:46) [thirdPartyAudit] Scanned 1378 class file(s) for forbidden API invocations (in 5.07s), 15 error(s). [delete] Deleting directory C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\tmp\thirdPartyAudit :benchmarks:thirdPartyAudit (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 6.439 secs. :benchmarks:precommit (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :benchmarks:precommit Skipping task ':benchmarks:precommit' as it has no actions. :benchmarks:precommit (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :build-tools:copyCheckstyleConf (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:copyCheckstyleConf Task ':build-tools:copyCheckstyleConf' is not up-to-date because: No history is available. :build-tools:copyCheckstyleConf (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.004 secs. :build-tools:checkstyleMain (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:checkstyleMain Task ':build-tools:checkstyleMain' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 4 files :build-tools:checkstyleMain (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.338 secs. :build-tools:compileTestJava (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:compileTestJava Task ':build-tools:compileTestJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':build-tools:compileTestJava'. :build-tools:compileTestJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\tmp\compileTestJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.134 secs. Created jar classpath snapshot for incremental compilation in 0.018 secs. 1556 duplicate classes found in classpath (see all with --debug). Written jar classpath snapshot for incremental compilation in 0.0 secs. :build-tools:compileTestJava (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 2.379 secs. :build-tools:compileTestGroovy (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:compileTestGroovy Task ':build-tools:compileTestGroovy' is not up-to-date because: No history is available. :build-tools:compileTestGroovy (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.501 secs. :build-tools:processTestResources (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:processTestResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\src\test\resources', not found Skipping task ':build-tools:processTestResources' as it has no source files and no previous output files. :build-tools:processTestResources (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :build-tools:testClasses (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:testClasses Skipping task ':build-tools:testClasses' as it has no actions. :build-tools:testClasses (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :build-tools:checkstyleTest (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:checkstyleTest Task ':build-tools:checkstyleTest' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 1 files :build-tools:checkstyleTest (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.315 secs. :build-tools:checkstyle (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:checkstyle Skipping task ':build-tools:checkstyle' as it has no actions. :build-tools:checkstyle (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :build-tools:dependencyLicenses (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:dependencyLicenses SKIPPED Skipping task ':build-tools:dependencyLicenses' as task onlyIf is false. :build-tools:dependencyLicenses (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :build-tools:filepermissions (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:filepermissions SKIPPED Skipping task ':build-tools:filepermissions' as task onlyIf is false. :build-tools:filepermissions (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :build-tools:forbiddenApisMain (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:forbiddenApisMain SKIPPED Skipping task ':build-tools:forbiddenApisMain' as task onlyIf is false. :build-tools:forbiddenApisMain (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :build-tools:forbiddenApisTest (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:forbiddenApisTest SKIPPED Skipping task ':build-tools:forbiddenApisTest' as task onlyIf is false. :build-tools:forbiddenApisTest (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :build-tools:forbiddenApis (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:forbiddenApis UP-TO-DATE Skipping task ':build-tools:forbiddenApis' as it has no actions. :build-tools:forbiddenApis (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :build-tools:forbiddenPatterns (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:forbiddenPatterns Task ':build-tools:forbiddenPatterns' is not up-to-date because: No history is available. file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\src\test\resources', not found :build-tools:forbiddenPatterns (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.144 secs. :build-tools:jarHell (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:jarHell SKIPPED Skipping task ':build-tools:jarHell' as task onlyIf is false. :build-tools:jarHell (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :build-tools:licenseHeaders (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:licenseHeaders SKIPPED Skipping task ':build-tools:licenseHeaders' as task onlyIf is false. :build-tools:licenseHeaders (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :build-tools:namingConventions (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:namingConventions Task ':build-tools:namingConventions' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc Command: C:\Users\jenkins\.java\java8\bin\java -Djna.nosys=true -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\distributions\build-tools-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\classes\java\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\classes\groovy\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\resources\main;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\4.7\generated-gradle-jars\gradle-api-4.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\wrapper\dists\gradle-4.7-all\4cret0dgl5o3b21weaoncl7ys\gradle-4.7\lib\groovy-all-2.4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\wrapper\dists\gradle-4.7-all\4cret0dgl5o3b21weaoncl7ys\gradle-4.7\lib\gradle-installation-beacon-4.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\junit4-ant\2.5.2\b8f91682cfeb8f9196aad56ace9c9a13330acef6\junit4-ant-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.netflix.nebula\gradle-extra-configurations-plugin\3.0.3\32cd905383b257b4fbd494807520c0b17617d974\gradle-extra-configurations-plugin-3.0.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.netflix.nebula\nebula-publishing-plugin\4.4.4\7dbb6e9eaa6f84680bbf98f2c995da26d907ade7\nebula-publishing-plugin-4.4.4.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.netflix.nebula\gradle-info-plugin\3.0.3\26507c144b8622489024282c6dcdb2e96f8f50da\gradle-info-plugin-3.0.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.eclipse.jgit\org.eclipse.jgit\3.2.0.201312181205-r\4b99546e8c8a04597b7a4564003e3b554ec12b5c\org.eclipse.jgit-3.2.0.201312181205-r.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.perforce\p4java\2012.3.551082\def6c8ac6206dc6fba82f579d4d5b56a4fd2836d\p4java-2012.3.551082.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\de.thetaphi\forbiddenapis\2.5\98dcf1ee01f4b353e1988f042d27e154c9add9b\forbiddenapis-2.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.rat\apache-rat\0.11\ad98b825264ab1d24a5b5bdc688e4a0226847250\apache-rat-0.11.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\ru.vyarus\gradle-animalsniffer-plugin\1.2.0\2858c5218f1116b2513554346596a05fa6c58c70\gradle-animalsniffer-plugin-1.2.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.gradle\gradle-logging\4.3\d2e47d6a87e11fceddaeb08bbcf7fe1e79e10621\gradle-logging-4.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.netflix.nebula\nebula-core\3.0.0\4b3277899a3e5b69189b938061b554f07cc8205b\nebula-core-3.0.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.perforce\p4java-jfrog\2011.1.297684\1de41848e31c6428e6eb7a71e8f88b9561d35fe1\p4java-jfrog-2011.1.297684.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.netflix.nebula\gradle-contacts-plugin\2.2.0\b8b1480722ff7c7ebcc408a641476a5fda341454\gradle-contacts-plugin-2.2.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.tmatesoft.svnkit\svnkit\1.8.5\a9600329d7e6271ab8ab7eb769c4df5adcdf503c\svnkit-1.8.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jsch\0.1.46\f3a38fd1d65c32bd627b1389e7181eb294cc8a8\jsch-0.1.46.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.googlecode.javaewah\JavaEWAH\0.5.6\1207c0fc8552d4f5f574b50f29321d923521128e\JavaEWAH-0.5.6.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jzlib\1.1.1\a1551373315ffc2f96130a0e5704f74e151777ba\jzlib-1.1.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.rat\apache-rat-tasks\0.11\ad888472d2861502b1651093e3a41685cce34cfd\apache-rat-tasks-0.11.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.rat\apache-rat-core\0.11\c02fdc6c4fca355491c429c8134a66bf66ccdc7e\apache-rat-core-0.11.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.commons\commons-lang3\3.2.1\66f13681add50ca9e4546ffabafaaac7645db3cf\commons-lang3-3.2.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jsch.agentproxy.svnkit-trilead-ssh2\0.0.7\95015f649026507428aa2f37baba69ba8a86dc67\jsch.agentproxy.svnkit-trilead-ssh2-0.0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jsch.agentproxy.connector-factory\0.0.7\780e0312287dd4e4f3601d401a073a081e403f2d\jsch.agentproxy.connector-factory-0.0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jsch.agentproxy.usocket-jna\0.0.7\98cf51ca8d7882c203729ffa48e9b7be4c50b835\jsch.agentproxy.usocket-jna-0.0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jsch.agentproxy.pageant\0.0.7\b40c0d9b10d79a0683ca93e3f70360b3d8045b62\jsch.agentproxy.pageant-0.0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.java.dev.jna\platform\3.5.2\beac07d13858ef3697ceeab43897d70aeb5113c9\platform-3.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.java.dev.jna\jna\3.5.2\f1bf9fe267e17c1a8f7d1aa2c985e7fe81a06da6\jna-3.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.tmatesoft.sqljet\sqljet\1.1.10\ede7fbabd4c96d34e48fda0e8feced24c98cedca\sqljet-1.1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\de.regnis.q.sequence\sequence-library\1.0.2\531ae7f7a4eac050dbce729edae139c2e546464\sequence-library-1.0.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.trilead\trilead-ssh2\1.0.0-build217\bd26b6a96923d5112cd20f821a16b589742ce1aa\trilead-ssh2-1.0.0-build217.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-collections\commons-collections\3.2.1\761ea405b9b37ced573d2df0d1e3a4e0f9edc668\commons-collections-3.2.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-lang\commons-lang\2.6\ce1edb914c94ebc388f086c6827e8bdeec71ac2\commons-lang-2.6.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-io\commons-io\2.2\83b5b8a7ba1c08f9e8c8ff2373724e33d3c1e22a\commons-io-2.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.commons\commons-compress\1.5\d2bd2c0bd328f1dabdf33e10b6d223ebcbe93343\commons-compress-1.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-cli\commons-cli\1.2\2bf96b7aa8b611c177d329452af1dc933e14501c\commons-cli-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jsch.agentproxy.usocket-nc\0.0.7\aff6f0f3a68f9731580d7c48075fdf177532dfe6\jsch.agentproxy.usocket-nc-0.0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jsch.agentproxy.sshagent\0.0.7\5c09a3d2a0562de53cf616794f43f51a3c0a496c\jsch.agentproxy.sshagent-0.0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jsch.agentproxy.core\0.0.7\9bcf818dc9e5247b01b1b18671a12caac293ae40\jsch.agentproxy.core-0.0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.antlr\antlr-runtime\3.4\8f011408269a8e42b8548687e137d8eeb56df4b4\antlr-runtime-3.4.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\classes\groovy\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\resources\test org.elasticsearch.test.NamingConventionsCheck --test-class org.elasticsearch.test.NamingConventionsCheckBadClasses$UnitTestCase --integ-test-class org.elasticsearch.test.NamingConventionsCheckBadClasses$IntegTestCase --self-test -- C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\classes\java\test Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' :build-tools:namingConventions (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.271 secs. :build-tools:namingConventionsMain (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:namingConventionsMain Task ':build-tools:namingConventionsMain' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc Command: C:\Users\jenkins\.java\java8\bin\java -Djna.nosys=true -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\distributions\build-tools-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\classes\java\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\classes\groovy\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\resources\main;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\4.7\generated-gradle-jars\gradle-api-4.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\wrapper\dists\gradle-4.7-all\4cret0dgl5o3b21weaoncl7ys\gradle-4.7\lib\groovy-all-2.4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\wrapper\dists\gradle-4.7-all\4cret0dgl5o3b21weaoncl7ys\gradle-4.7\lib\gradle-installation-beacon-4.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\junit4-ant\2.5.2\b8f91682cfeb8f9196aad56ace9c9a13330acef6\junit4-ant-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.netflix.nebula\gradle-extra-configurations-plugin\3.0.3\32cd905383b257b4fbd494807520c0b17617d974\gradle-extra-configurations-plugin-3.0.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.netflix.nebula\nebula-publishing-plugin\4.4.4\7dbb6e9eaa6f84680bbf98f2c995da26d907ade7\nebula-publishing-plugin-4.4.4.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.netflix.nebula\gradle-info-plugin\3.0.3\26507c144b8622489024282c6dcdb2e96f8f50da\gradle-info-plugin-3.0.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.eclipse.jgit\org.eclipse.jgit\3.2.0.201312181205-r\4b99546e8c8a04597b7a4564003e3b554ec12b5c\org.eclipse.jgit-3.2.0.201312181205-r.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.perforce\p4java\2012.3.551082\def6c8ac6206dc6fba82f579d4d5b56a4fd2836d\p4java-2012.3.551082.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\de.thetaphi\forbiddenapis\2.5\98dcf1ee01f4b353e1988f042d27e154c9add9b\forbiddenapis-2.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.rat\apache-rat\0.11\ad98b825264ab1d24a5b5bdc688e4a0226847250\apache-rat-0.11.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\ru.vyarus\gradle-animalsniffer-plugin\1.2.0\2858c5218f1116b2513554346596a05fa6c58c70\gradle-animalsniffer-plugin-1.2.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.gradle\gradle-logging\4.3\d2e47d6a87e11fceddaeb08bbcf7fe1e79e10621\gradle-logging-4.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.netflix.nebula\nebula-core\3.0.0\4b3277899a3e5b69189b938061b554f07cc8205b\nebula-core-3.0.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.perforce\p4java-jfrog\2011.1.297684\1de41848e31c6428e6eb7a71e8f88b9561d35fe1\p4java-jfrog-2011.1.297684.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.netflix.nebula\gradle-contacts-plugin\2.2.0\b8b1480722ff7c7ebcc408a641476a5fda341454\gradle-contacts-plugin-2.2.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.tmatesoft.svnkit\svnkit\1.8.5\a9600329d7e6271ab8ab7eb769c4df5adcdf503c\svnkit-1.8.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jsch\0.1.46\f3a38fd1d65c32bd627b1389e7181eb294cc8a8\jsch-0.1.46.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.googlecode.javaewah\JavaEWAH\0.5.6\1207c0fc8552d4f5f574b50f29321d923521128e\JavaEWAH-0.5.6.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jzlib\1.1.1\a1551373315ffc2f96130a0e5704f74e151777ba\jzlib-1.1.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.rat\apache-rat-tasks\0.11\ad888472d2861502b1651093e3a41685cce34cfd\apache-rat-tasks-0.11.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.rat\apache-rat-core\0.11\c02fdc6c4fca355491c429c8134a66bf66ccdc7e\apache-rat-core-0.11.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.commons\commons-lang3\3.2.1\66f13681add50ca9e4546ffabafaaac7645db3cf\commons-lang3-3.2.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jsch.agentproxy.svnkit-trilead-ssh2\0.0.7\95015f649026507428aa2f37baba69ba8a86dc67\jsch.agentproxy.svnkit-trilead-ssh2-0.0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jsch.agentproxy.connector-factory\0.0.7\780e0312287dd4e4f3601d401a073a081e403f2d\jsch.agentproxy.connector-factory-0.0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jsch.agentproxy.usocket-jna\0.0.7\98cf51ca8d7882c203729ffa48e9b7be4c50b835\jsch.agentproxy.usocket-jna-0.0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jsch.agentproxy.pageant\0.0.7\b40c0d9b10d79a0683ca93e3f70360b3d8045b62\jsch.agentproxy.pageant-0.0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.java.dev.jna\platform\3.5.2\beac07d13858ef3697ceeab43897d70aeb5113c9\platform-3.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.java.dev.jna\jna\3.5.2\f1bf9fe267e17c1a8f7d1aa2c985e7fe81a06da6\jna-3.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.tmatesoft.sqljet\sqljet\1.1.10\ede7fbabd4c96d34e48fda0e8feced24c98cedca\sqljet-1.1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\de.regnis.q.sequence\sequence-library\1.0.2\531ae7f7a4eac050dbce729edae139c2e546464\sequence-library-1.0.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.trilead\trilead-ssh2\1.0.0-build217\bd26b6a96923d5112cd20f821a16b589742ce1aa\trilead-ssh2-1.0.0-build217.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-collections\commons-collections\3.2.1\761ea405b9b37ced573d2df0d1e3a4e0f9edc668\commons-collections-3.2.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-lang\commons-lang\2.6\ce1edb914c94ebc388f086c6827e8bdeec71ac2\commons-lang-2.6.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-io\commons-io\2.2\83b5b8a7ba1c08f9e8c8ff2373724e33d3c1e22a\commons-io-2.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.commons\commons-compress\1.5\d2bd2c0bd328f1dabdf33e10b6d223ebcbe93343\commons-compress-1.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-cli\commons-cli\1.2\2bf96b7aa8b611c177d329452af1dc933e14501c\commons-cli-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jsch.agentproxy.usocket-nc\0.0.7\aff6f0f3a68f9731580d7c48075fdf177532dfe6\jsch.agentproxy.usocket-nc-0.0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jsch.agentproxy.sshagent\0.0.7\5c09a3d2a0562de53cf616794f43f51a3c0a496c\jsch.agentproxy.sshagent-0.0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.jcraft\jsch.agentproxy.core\0.0.7\9bcf818dc9e5247b01b1b18671a12caac293ae40\jsch.agentproxy.core-0.0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.antlr\antlr-runtime\3.4\8f011408269a8e42b8548687e137d8eeb56df4b4\antlr-runtime-3.4.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\classes\groovy\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\resources\test org.elasticsearch.test.NamingConventionsCheck --test-class org.elasticsearch.test.NamingConventionsCheckBadClasses$UnitTestCase --integ-test-class org.elasticsearch.test.NamingConventionsCheckBadClasses$IntegTestCase --self-test --main -- C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\classes\java\main Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' :build-tools:namingConventionsMain (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.258 secs. :build-tools:thirdPartyAudit (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:thirdPartyAudit SKIPPED Skipping task ':build-tools:thirdPartyAudit' as task onlyIf is false. :build-tools:thirdPartyAudit (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :build-tools:precommit (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:precommit Skipping task ':build-tools:precommit' as it has no actions. :build-tools:precommit (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :build-tools:test (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :build-tools:test SKIPPED Skipping task ':build-tools:test' as task onlyIf is false. :build-tools:test (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :server:copyCheckstyleConf (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :server:copyCheckstyleConf Task ':server:copyCheckstyleConf' is not up-to-date because: No history is available. :server:copyCheckstyleConf (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.003 secs. :server:checkstyleJava9 (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :server:checkstyleJava9 Task ':server:checkstyleJava9' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 1 files :server:checkstyleJava9 (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.223 secs. :server:checkstyleMain (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :server:checkstyleMain Task ':server:checkstyleMain' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 2878 files :server:checkstyleMain (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 20.165 secs. :client:rest:compileJava (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :client:rest:compileJava Task ':client:rest:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':client:rest:compileJava'. :client:rest:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.012 secs. Created jar classpath snapshot for incremental compilation in 0.341 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :client:rest:compileJava (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 2.045 secs. :client:rest:processResources (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :client:rest:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\src\main\resources', not found Skipping task ':client:rest:processResources' as it has no source files and no previous output files. :client:rest:processResources (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :client:rest:classes (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :client:rest:classes Skipping task ':client:rest:classes' as it has no actions. :client:rest:classes (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :client:rest:jar (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :client:rest:jar Task ':client:rest:jar' is not up-to-date because: No history is available. :client:rest:jar (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.028 secs. :libs:elasticsearch-nio:compileJava (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :libs:elasticsearch-nio:compileJava Task ':libs:elasticsearch-nio:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':libs:elasticsearch-nio:compileJava'. :libs:elasticsearch-nio:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.011 secs. Created jar classpath snapshot for incremental compilation in 0.0 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :libs:elasticsearch-nio:compileJava (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 1.772 secs. :libs:elasticsearch-nio:processResources (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :libs:elasticsearch-nio:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\src\main\resources', not found Skipping task ':libs:elasticsearch-nio:processResources' as it has no source files and no previous output files. :libs:elasticsearch-nio:processResources (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :libs:elasticsearch-nio:classes (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :libs:elasticsearch-nio:classes Skipping task ':libs:elasticsearch-nio:classes' as it has no actions. :libs:elasticsearch-nio:classes (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.0 secs. :libs:elasticsearch-nio:jar (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :libs:elasticsearch-nio:jar Task ':libs:elasticsearch-nio:jar' is not up-to-date because: No history is available. :libs:elasticsearch-nio:jar (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.022 secs. :test:framework:compileJava (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :test:framework:compileJava Task ':test:framework:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':test:framework:compileJava'. :test:framework:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. Class dependency analysis for incremental compilation took 0.167 secs. Created jar classpath snapshot for incremental compilation in 0.473 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :test:framework:compileJava (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 8.708 secs. :test:framework:processResources (Thread[Task worker for ':' Thread 2,5,main]) started. > Task :test:framework:processResources Task ':test:framework:processResources' is not up-to-date because: No history is available. :test:framework:processResources (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 0.014 secs. :test:framework:classes (Thread[Task worker for ':',5,main]) started. > Task :test:framework:classes Skipping task ':test:framework:classes' as it has no actions. :test:framework:classes (Thread[Task worker for ':',5,main]) completed. Took 0.0 secs. :test:framework:jar (Thread[Task worker for ':',5,main]) started. > Task :test:framework:jar Task ':test:framework:jar' is not up-to-date because: No history is available. :test:framework:jar (Thread[Task worker for ':',5,main]) completed. Took 0.243 secs. :server:compileTestJava (Thread[Task worker for ':',5,main]) started. > Task :server:compileTestJava Task ':server:compileTestJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':server:compileTestJava'. :server:compileTestJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\compileTestJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. Class dependency analysis for incremental compilation took 1.543 secs. Created jar classpath snapshot for incremental compilation in 0.461 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :server:compileTestJava (Thread[Task worker for ':',5,main]) completed. Took 57.773 secs. :server:processTestResources (Thread[Task worker for ':',5,main]) started. > Task :server:processTestResources Task ':server:processTestResources' is not up-to-date because: No history is available. :server:processTestResources (Thread[Task worker for ':',5,main]) completed. Took 0.529 secs. :server:testClasses (Thread[Task worker for ':',5,main]) started. > Task :server:testClasses Skipping task ':server:testClasses' as it has no actions. :server:testClasses (Thread[Task worker for ':',5,main]) completed. Took 0.0 secs. :server:checkstyleTest (Thread[Task worker for ':',5,main]) started. > Task :server:checkstyleTest Task ':server:checkstyleTest' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 1447 files :server:checkstyleTest (Thread[Task worker for ':',5,main]) completed. Took 21.358 secs. :server:checkstyle (Thread[Task worker for ':',5,main]) started. > Task :server:checkstyle Skipping task ':server:checkstyle' as it has no actions. :server:checkstyle (Thread[Task worker for ':',5,main]) completed. Took 0.0 secs. :server:dependencyLicenses (Thread[Task worker for ':',5,main]) started. > Task :server:dependencyLicenses Task ':server:dependencyLicenses' is not up-to-date because: Task has not declared any outputs despite executing actions. Checking sha for lucene-core-7.4.0-snapshot-1ed95c097b.jar Checking license/notice for lucene-core Mapped dependency name lucene-core to lucene for license check Checking sha for lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar Checking license/notice for lucene-analyzers-common Mapped dependency name lucene-analyzers-common to lucene for license check Checking sha for lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar Checking license/notice for lucene-backward-codecs Mapped dependency name lucene-backward-codecs to lucene for license check Checking sha for lucene-grouping-7.4.0-snapshot-1ed95c097b.jar Checking license/notice for lucene-grouping Mapped dependency name lucene-grouping to lucene for license check Checking sha for lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar Checking license/notice for lucene-highlighter Mapped dependency name lucene-highlighter to lucene for license check Checking sha for lucene-join-7.4.0-snapshot-1ed95c097b.jar Checking license/notice for lucene-join Mapped dependency name lucene-join to lucene for license check Checking sha for lucene-memory-7.4.0-snapshot-1ed95c097b.jar Checking license/notice for lucene-memory Mapped dependency name lucene-memory to lucene for license check Checking sha for lucene-misc-7.4.0-snapshot-1ed95c097b.jar Checking license/notice for lucene-misc Mapped dependency name lucene-misc to lucene for license check Checking sha for lucene-queries-7.4.0-snapshot-1ed95c097b.jar Checking license/notice for lucene-queries Mapped dependency name lucene-queries to lucene for license check Checking sha for lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar Checking license/notice for lucene-queryparser Mapped dependency name lucene-queryparser to lucene for license check Checking sha for lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar Checking license/notice for lucene-sandbox Mapped dependency name lucene-sandbox to lucene for license check Checking sha for lucene-spatial-7.4.0-snapshot-1ed95c097b.jar Checking license/notice for lucene-spatial Mapped dependency name lucene-spatial to lucene for license check Checking sha for lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar Checking license/notice for lucene-spatial-extras Mapped dependency name lucene-spatial-extras to lucene for license check Checking sha for lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar Checking license/notice for lucene-spatial3d Mapped dependency name lucene-spatial3d to lucene for license check Checking sha for lucene-suggest-7.4.0-snapshot-1ed95c097b.jar Checking license/notice for lucene-suggest Mapped dependency name lucene-suggest to lucene for license check Checking sha for hppc-0.7.1.jar Checking license/notice for hppc Checking sha for joda-time-2.9.9.jar Checking license/notice for joda-time Checking sha for t-digest-3.2.jar Checking license/notice for t-digest Checking sha for HdrHistogram-2.1.9.jar Checking license/notice for HdrHistogram Checking sha for spatial4j-0.7.jar Checking license/notice for spatial4j Checking sha for jts-core-1.15.0.jar Checking license/notice for jts-core Checking sha for log4j-api-2.9.1.jar Checking license/notice for log4j-api Checking sha for log4j-core-2.9.1.jar Checking license/notice for log4j-core Checking sha for log4j-1.2-api-2.9.1.jar Checking license/notice for log4j Checking sha for jna-4.5.1.jar Checking license/notice for jna :server:dependencyLicenses (Thread[Task worker for ':',5,main]) completed. Took 0.132 secs. :server:filepermissions (Thread[Daemon worker,5,main]) started. > Task :server:filepermissions SKIPPED Skipping task ':server:filepermissions' as task onlyIf is false. :server:filepermissions (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :server:forbiddenApisMain (Thread[Daemon worker,5,main]) started. > Task :server:forbiddenApisMain Task ':server:forbiddenApisMain' is not up-to-date because: No history is available. Reading bundled API signatures: jdk-unsafe-1.8 Reading bundled API signatures: jdk-deprecated-1.8 Reading bundled API signatures: jdk-non-portable Reading bundled API signatures: jdk-system-out Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/jdk-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-all-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-server-signatures.txt Loading classes to check... Scanning classes for violations... Scanned 5441 class file(s) for forbidden API invocations (in 4.14s), 0 error(s). :server:forbiddenApisMain (Thread[Daemon worker,5,main]) completed. Took 4.49 secs. :server:forbiddenApisTest (Thread[Daemon worker,5,main]) started. > Task :server:forbiddenApisTest Task ':server:forbiddenApisTest' is not up-to-date because: No history is available. Reading bundled API signatures: jdk-unsafe-1.8 Reading bundled API signatures: jdk-deprecated-1.8 Reading bundled API signatures: jdk-non-portable Reading bundled API signatures: jdk-system-out Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/jdk-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-all-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-test-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/http-signatures.txt Loading classes to check... Scanning classes for violations... Scanned 2923 class file(s) for forbidden API invocations (in 4.64s), 0 error(s). :server:forbiddenApisTest (Thread[Daemon worker,5,main]) completed. Took 4.898 secs. :server:forbiddenApis (Thread[Daemon worker,5,main]) started. > Task :server:forbiddenApis Skipping task ':server:forbiddenApis' as it has no actions. :server:forbiddenApis (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :server:forbiddenPatterns (Thread[Daemon worker,5,main]) started. > Task :server:forbiddenPatterns Task ':server:forbiddenPatterns' is not up-to-date because: No history is available. file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\src\java9\resources', not found :server:forbiddenPatterns (Thread[Daemon worker,5,main]) completed. Took 3.072 secs. :server:jarHell (Thread[Daemon worker,5,main]) started. > Task :server:jarHell Task ':server:jarHell' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server Command: C:\Users\jenkins\.java\java8\bin\java -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\resources\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\classes\java\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\resources\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader\build\distributions\plugin-classloader-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.google.jimfs\jimfs\1.1\8fbd0579dc68aba6186935cc1bee21d2f3e7ec1c\jimfs-1.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.google.guava\guava\18.0\cce0823396aa693798f8882e64213b1772032b09\guava-18.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar org.elasticsearch.bootstrap.JarHell Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' checking for jar hell... java.class.path: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\resources\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\classes\java\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\resources\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader\build\distributions\plugin-classloader-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.google.jimfs\jimfs\1.1\8fbd0579dc68aba6186935cc1bee21d2f3e7ec1c\jimfs-1.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.google.guava\guava\18.0\cce0823396aa693798f8882e64213b1772032b09\guava-18.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar sun.boot.class.path: C:\Users\jenkins\.java\java8\jre\lib\resources.jar;C:\Users\jenkins\.java\java8\jre\lib\rt.jar;C:\Users\jenkins\.java\java8\jre\lib\sunrsasign.jar;C:\Users\jenkins\.java\java8\jre\lib\jsse.jar;C:\Users\jenkins\.java\java8\jre\lib\jce.jar;C:\Users\jenkins\.java\java8\jre\lib\charsets.jar;C:\Users\jenkins\.java\java8\jre\lib\jfr.jar;C:\Users\jenkins\.java\java8\jre\classes classloader urls: [file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/server/build/classes/java/test/, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/server/build/resources/test/, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/server/build/classes/java/main/, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/server/build/resources/main/, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/x-content/build/distributions/elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/test/framework/build/distributions/framework-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/server/cli/build/distributions/elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/elasticsearch-core/build/distributions/elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/secure-sm/build/distributions/elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/7.4.0-snapshot-1ed95c097b/b06e4f6514256a3f187a9892e520638b9c59e63/lucene-core-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/7.4.0-snapshot-1ed95c097b/2b2be48f6622c150496e755497e7bdb8daa46030/lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-backward-codecs/7.4.0-snapshot-1ed95c097b/6cbafc48e8ac4966377665eb3bbe93f9addf04a5/lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-grouping/7.4.0-snapshot-1ed95c097b/4c71cef87fe513a7a96c2a7980ed6f7c2b015763/lucene-grouping-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-highlighter/7.4.0-snapshot-1ed95c097b/665e044d1180100940bccd7e8e41dde48e342da3/lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-join/7.4.0-snapshot-1ed95c097b/d343bbf5792f5969288b59b51179acd29d04f4ee/lucene-join-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-memory/7.4.0-snapshot-1ed95c097b/8915f3c93af3348655bcc204289f9011835738a2/lucene-memory-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-misc/7.4.0-snapshot-1ed95c097b/e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d/lucene-misc-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queries/7.4.0-snapshot-1ed95c097b/5946d5e2be276f66e9ff6d6111acabb03a9330d9/lucene-queries-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/7.4.0-snapshot-1ed95c097b/d9fc5fc63f3d861e5af72e11373368e8a4c6bba6/lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/7.4.0-snapshot-1ed95c097b/ee283c0a1a717f3e0915de75864a93d043efaee3/lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial/7.4.0-snapshot-1ed95c097b/e1adf0220a7c052ac81e2919ffac24ac0e5b007c/lucene-spatial-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial-extras/7.4.0-snapshot-1ed95c097b/6d9306053942c48f43392a634f11a95462b5996e/lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial3d/7.4.0-snapshot-1ed95c097b/2334e8c5f4d0f98659b30e0c2035296e4aae8ff5/lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-suggest/7.4.0-snapshot-1ed95c097b/f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463/lucene-suggest-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.carrotsearch/hppc/0.7.1/8b5057f74ea378c0150a1860874a3ebdcb713767/hppc-0.7.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/joda-time/joda-time/2.9.9/f7b520c458572890807d143670c9b24f4de90897/joda-time-2.9.9.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.tdunning/t-digest/3.2/2ab94758b0276a8a26102adf8d528cf6d0567b9a/t-digest-3.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.hdrhistogram/HdrHistogram/2.1.9/e4631ce165eb400edecfa32e03d3f1be53dee754/HdrHistogram-2.1.9.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.locationtech.spatial4j/spatial4j/0.7/faa8ba85d503da4ab872d17ba8c00da0098ab2f2/spatial4j-0.7.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.locationtech.jts/jts-core/1.15.0/705981b7e25d05a76a3654e597dab6ba423eb79e/jts-core-1.15.0.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/elasticsearch-nio/build/distributions/elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.9.1/7a2999229464e7a324aa503c0a52ec0f05efe7bd/log4j-api-2.9.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.9.1/c041978c686866ee8534f538c6220238db3bb6be/log4j-core-2.9.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-1.2-api/2.9.1/894f96d677880d4ab834a1356f62b875e579caaa/log4j-1.2-api-2.9.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.elasticsearch/jna/4.5.1/da10908ae23dc59b19dc258e63aea1c44621dc3a/jna-4.5.1.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/plugin-classloader/build/distributions/plugin-classloader-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.google.jimfs/jimfs/1.1/8fbd0579dc68aba6186935cc1bee21d2f3e7ec1c/jimfs-1.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/18.0/cce0823396aa693798f8882e64213b1772032b09/guava-18.0.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.yaml/snakeyaml/1.17/7a27ea250c5130b2922b86dea63cbb1cc10a660c/snakeyaml-1.17.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.8.10/eb21a035c66ad307e66ec8fce37f5d50fd62d039/jackson-core-2.8.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-smile/2.8.10/e853081fadaad3e98ed801937acc3d8f77580686/jackson-dataformat-smile-2.8.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-yaml/2.8.10/1e08caf1d787c825307d8cc6362452086020d853/jackson-dataformat-yaml-2.8.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-cbor/2.8.10/1c58cc9313ddf19f0900cd61ed044874278ce320/jackson-dataformat-cbor-2.8.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/5.0.2/98cafc6081d5632b61be2c9e60650b64ddbc637c/jopt-simple-5.0.2.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/client/rest/build/distributions/elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.carrotsearch.randomizedtesting/randomizedtesting-runner/2.5.2/91f3284993b44dcb2f003b5f28617abba13971d2/randomizedtesting-runner-2.5.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/junit/junit/4.12/2973d150c0dc1fefe998f834810d68f278ea58ec/junit-4.12.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-all/1.3/63a21ebc981131004ad02e0434e799fd7f3a8d5a/hamcrest-all-1.3.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-test-framework/7.4.0-snapshot-1ed95c097b/1afd168832ee0cf36d1adcc960c3e127d8848bd0/lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-codecs/7.4.0-snapshot-1ed95c097b/4ad08cbda4541cbf5899860aa99c08e557585f74/lucene-codecs-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/commons-logging/commons-logging/1.1.3/f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f/commons-logging-1.1.3.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/commons-codec/commons-codec/1.10/4b95f4897fa13f2cd904aee711aeafc0c5295cd8/commons-codec-1.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.elasticsearch/securemock/1.2/98201d4ad5ac93f6b415ae9172d52b5e7cda490e/securemock-1.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.elasticsearch/mocksocket/1.2/190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361/mocksocket-1.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpclient/4.5.2/733db77aa8d9b2d68015189df76ab06304406e50/httpclient-4.5.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpcore/4.4.5/e7501a1b34325abb00d17dde96150604a0658b54/httpcore-4.4.5.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpasyncclient/4.1.2/95aa3e6fb520191a0970a73cf09f62948ee614be/httpasyncclient-4.1.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpcore-nio/4.4.5/f4be009e7505f6ceddf21e7960c759f413f15056/httpcore-nio-4.4.5.jar] java.home: C:\Users\jenkins\.java\java8\jre examining directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\classes\java\test examining directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\resources\test examining directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\classes\java\main examining directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\resources\main examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader\build\distributions\plugin-classloader-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.google.jimfs\jimfs\1.1\8fbd0579dc68aba6186935cc1bee21d2f3e7ec1c\jimfs-1.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.google.guava\guava\18.0\cce0823396aa693798f8882e64213b1772032b09\guava-18.0.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar no jar hell found :server:jarHell (Thread[Daemon worker,5,main]) completed. Took 0.812 secs. :server:licenseHeaders (Thread[Daemon worker,5,main]) started. > Task :server:licenseHeaders Task ':server:licenseHeaders' is not up-to-date because: No history is available. :server:licenseHeaders (Thread[Daemon worker,5,main]) completed. Took 1.497 secs. :server:loggerUsageCheck (Thread[Daemon worker,5,main]) started. > Task :server:loggerUsageCheck Task ':server:loggerUsageCheck' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server Command: C:\Users\jenkins\.java\java8\bin\java -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\distributions\logger-usage-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.ow2.asm\asm-debug-all\5.0.4\702b8525fcf81454235e5e2fa2a35f15ffc0ec7e\asm-debug-all-5.0.4.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar org.elasticsearch.test.loggerusage.ESLoggerUsageChecker C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\classes\java\main C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\classes\java\test Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' checking for wrong usages of ESLogger... No wrong usages found :server:loggerUsageCheck (Thread[Daemon worker,5,main]) completed. Took 6.164 secs. :server:namingConventions (Thread[Daemon worker,5,main]) started. > Task :server:namingConventions Task ':server:namingConventions' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server Command: C:\Users\jenkins\.java\java8\bin\java -Djna.nosys=true -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\distributions\build-tools-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\classes\java\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\resources\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader\build\distributions\plugin-classloader-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.google.jimfs\jimfs\1.1\8fbd0579dc68aba6186935cc1bee21d2f3e7ec1c\jimfs-1.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.google.guava\guava\18.0\cce0823396aa693798f8882e64213b1772032b09\guava-18.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\resources\test org.elasticsearch.test.NamingConventionsCheck --test-class org.apache.lucene.util.LuceneTestCase --integ-test-class org.elasticsearch.test.ESIntegTestCase -- C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\classes\java\test Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' :server:namingConventions (Thread[Daemon worker,5,main]) completed. Took 1.826 secs. :server:thirdPartyAudit (Thread[Daemon worker,5,main]) started. > Task :server:thirdPartyAudit Task ':server:thirdPartyAudit' is not up-to-date because: No history is available. [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit [thirdPartyAudit] Reading inline API signatures... [thirdPartyAudit] Loading classes to check... [thirdPartyAudit] Scanning classes for violations... [thirdPartyAudit] Scanned 6899 class file(s) for forbidden API invocations (in 7.85s), 0 error(s). [delete] Deleting directory C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\thirdPartyAudit :server:thirdPartyAudit (Thread[Daemon worker,5,main]) completed. Took 13.917 secs. :server:precommit (Thread[Daemon worker,5,main]) started. > Task :server:precommit Skipping task ':server:precommit' as it has no actions. :server:precommit (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:elasticsearch-core:copyCheckstyleConf (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:copyCheckstyleConf Task ':libs:elasticsearch-core:copyCheckstyleConf' is not up-to-date because: No history is available. :libs:elasticsearch-core:copyCheckstyleConf (Thread[Daemon worker,5,main]) completed. Took 0.003 secs. :libs:elasticsearch-core:checkstyleJava9 (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:checkstyleJava9 Task ':libs:elasticsearch-core:checkstyleJava9' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 1 files :libs:elasticsearch-core:checkstyleJava9 (Thread[Daemon worker,5,main]) completed. Took 0.079 secs. :libs:elasticsearch-core:checkstyleMain (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:checkstyleMain Task ':libs:elasticsearch-core:checkstyleMain' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 12 files :libs:elasticsearch-core:checkstyleMain (Thread[Daemon worker,5,main]) completed. Took 0.174 secs. :libs:elasticsearch-core:compileTestJava (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:compileTestJava Task ':libs:elasticsearch-core:compileTestJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':libs:elasticsearch-core:compileTestJava'. :libs:elasticsearch-core:compileTestJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\tmp\compileTestJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.009 secs. Created jar classpath snapshot for incremental compilation in 0.005 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :libs:elasticsearch-core:compileTestJava (Thread[Daemon worker,5,main]) completed. Took 2.102 secs. :libs:elasticsearch-core:processTestResources (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:processTestResources Task ':libs:elasticsearch-core:processTestResources' is not up-to-date because: No history is available. :libs:elasticsearch-core:processTestResources (Thread[Daemon worker,5,main]) completed. Took 0.006 secs. :libs:elasticsearch-core:testClasses (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:testClasses Skipping task ':libs:elasticsearch-core:testClasses' as it has no actions. :libs:elasticsearch-core:testClasses (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:elasticsearch-core:checkstyleTest (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:checkstyleTest Task ':libs:elasticsearch-core:checkstyleTest' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 5 files :libs:elasticsearch-core:checkstyleTest (Thread[Daemon worker,5,main]) completed. Took 0.148 secs. :libs:elasticsearch-core:checkstyle (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:checkstyle Skipping task ':libs:elasticsearch-core:checkstyle' as it has no actions. :libs:elasticsearch-core:checkstyle (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:elasticsearch-core:dependencyLicenses (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:dependencyLicenses Task ':libs:elasticsearch-core:dependencyLicenses' is not up-to-date because: Task has not declared any outputs despite executing actions. :libs:elasticsearch-core:dependencyLicenses (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:elasticsearch-core:filepermissions (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:filepermissions SKIPPED Skipping task ':libs:elasticsearch-core:filepermissions' as task onlyIf is false. :libs:elasticsearch-core:filepermissions (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:elasticsearch-core:forbiddenApisMain (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:forbiddenApisMain Task ':libs:elasticsearch-core:forbiddenApisMain' is not up-to-date because: No history is available. Reading bundled API signatures: jdk-unsafe-1.8 Reading bundled API signatures: jdk-deprecated-1.8 Reading bundled API signatures: jdk-non-portable Reading bundled API signatures: jdk-system-out Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/jdk-signatures.txt Loading classes to check... Scanning classes for violations... Scanned 15 class file(s) for forbidden API invocations (in 0.04s), 0 error(s). :libs:elasticsearch-core:forbiddenApisMain (Thread[Daemon worker,5,main]) completed. Took 0.042 secs. :libs:elasticsearch-core:forbiddenApisTest (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:forbiddenApisTest Task ':libs:elasticsearch-core:forbiddenApisTest' is not up-to-date because: No history is available. Reading bundled API signatures: jdk-unsafe-1.8 Reading bundled API signatures: jdk-deprecated-1.8 Reading bundled API signatures: jdk-non-portable Reading bundled API signatures: jdk-system-out Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/jdk-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-all-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-test-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/http-signatures.txt Loading classes to check... Scanning classes for violations... Scanned 6 class file(s) for forbidden API invocations (in 0.07s), 0 error(s). :libs:elasticsearch-core:forbiddenApisTest (Thread[Daemon worker,5,main]) completed. Took 0.086 secs. :libs:elasticsearch-core:forbiddenApis (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:forbiddenApis Skipping task ':libs:elasticsearch-core:forbiddenApis' as it has no actions. :libs:elasticsearch-core:forbiddenApis (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:elasticsearch-core:forbiddenPatterns (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:forbiddenPatterns Task ':libs:elasticsearch-core:forbiddenPatterns' is not up-to-date because: No history is available. file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\src\java9\resources', not found file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\src\main\resources', not found :libs:elasticsearch-core:forbiddenPatterns (Thread[Daemon worker,5,main]) completed. Took 0.021 secs. :libs:elasticsearch-core:jarHell (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:jarHell Task ':libs:elasticsearch-core:jarHell' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core Command: C:\Users\jenkins\.java\java8\bin\java -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\resources\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\classes\java\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar org.elasticsearch.bootstrap.JarHell Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' checking for jar hell... java.class.path: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\resources\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\classes\java\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar sun.boot.class.path: C:\Users\jenkins\.java\java8\jre\lib\resources.jar;C:\Users\jenkins\.java\java8\jre\lib\rt.jar;C:\Users\jenkins\.java\java8\jre\lib\sunrsasign.jar;C:\Users\jenkins\.java\java8\jre\lib\jsse.jar;C:\Users\jenkins\.java\java8\jre\lib\jce.jar;C:\Users\jenkins\.java\java8\jre\lib\charsets.jar;C:\Users\jenkins\.java\java8\jre\lib\jfr.jar;C:\Users\jenkins\.java\java8\jre\classes classloader urls: [file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/elasticsearch-core/build/classes/java/test/, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/elasticsearch-core/build/resources/test/, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/elasticsearch-core/build/classes/java/main/, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/test/framework/build/distributions/framework-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.carrotsearch.randomizedtesting/randomizedtesting-runner/2.5.2/91f3284993b44dcb2f003b5f28617abba13971d2/randomizedtesting-runner-2.5.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/junit/junit/4.12/2973d150c0dc1fefe998f834810d68f278ea58ec/junit-4.12.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-all/1.3/63a21ebc981131004ad02e0434e799fd7f3a8d5a/hamcrest-all-1.3.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/client/rest/build/distributions/elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/elasticsearch-nio/build/distributions/elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/server/build/distributions/elasticsearch-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/server/cli/build/distributions/elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-test-framework/7.4.0-snapshot-1ed95c097b/1afd168832ee0cf36d1adcc960c3e127d8848bd0/lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-codecs/7.4.0-snapshot-1ed95c097b/4ad08cbda4541cbf5899860aa99c08e557585f74/lucene-codecs-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/commons-logging/commons-logging/1.1.3/f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f/commons-logging-1.1.3.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/commons-codec/commons-codec/1.10/4b95f4897fa13f2cd904aee711aeafc0c5295cd8/commons-codec-1.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.elasticsearch/securemock/1.2/98201d4ad5ac93f6b415ae9172d52b5e7cda490e/securemock-1.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.elasticsearch/mocksocket/1.2/190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361/mocksocket-1.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpclient/4.5.2/733db77aa8d9b2d68015189df76ab06304406e50/httpclient-4.5.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpcore/4.4.5/e7501a1b34325abb00d17dde96150604a0658b54/httpcore-4.4.5.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpasyncclient/4.1.2/95aa3e6fb520191a0970a73cf09f62948ee614be/httpasyncclient-4.1.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpcore-nio/4.4.5/f4be009e7505f6ceddf21e7960c759f413f15056/httpcore-nio-4.4.5.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.9.1/7a2999229464e7a324aa503c0a52ec0f05efe7bd/log4j-api-2.9.1.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/secure-sm/build/distributions/elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/x-content/build/distributions/elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/7.4.0-snapshot-1ed95c097b/b06e4f6514256a3f187a9892e520638b9c59e63/lucene-core-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/7.4.0-snapshot-1ed95c097b/2b2be48f6622c150496e755497e7bdb8daa46030/lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-backward-codecs/7.4.0-snapshot-1ed95c097b/6cbafc48e8ac4966377665eb3bbe93f9addf04a5/lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-grouping/7.4.0-snapshot-1ed95c097b/4c71cef87fe513a7a96c2a7980ed6f7c2b015763/lucene-grouping-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-highlighter/7.4.0-snapshot-1ed95c097b/665e044d1180100940bccd7e8e41dde48e342da3/lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-join/7.4.0-snapshot-1ed95c097b/d343bbf5792f5969288b59b51179acd29d04f4ee/lucene-join-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-memory/7.4.0-snapshot-1ed95c097b/8915f3c93af3348655bcc204289f9011835738a2/lucene-memory-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-misc/7.4.0-snapshot-1ed95c097b/e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d/lucene-misc-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queries/7.4.0-snapshot-1ed95c097b/5946d5e2be276f66e9ff6d6111acabb03a9330d9/lucene-queries-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/7.4.0-snapshot-1ed95c097b/d9fc5fc63f3d861e5af72e11373368e8a4c6bba6/lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/7.4.0-snapshot-1ed95c097b/ee283c0a1a717f3e0915de75864a93d043efaee3/lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial/7.4.0-snapshot-1ed95c097b/e1adf0220a7c052ac81e2919ffac24ac0e5b007c/lucene-spatial-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial-extras/7.4.0-snapshot-1ed95c097b/6d9306053942c48f43392a634f11a95462b5996e/lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial3d/7.4.0-snapshot-1ed95c097b/2334e8c5f4d0f98659b30e0c2035296e4aae8ff5/lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-suggest/7.4.0-snapshot-1ed95c097b/f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463/lucene-suggest-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.carrotsearch/hppc/0.7.1/8b5057f74ea378c0150a1860874a3ebdcb713767/hppc-0.7.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/joda-time/joda-time/2.9.9/f7b520c458572890807d143670c9b24f4de90897/joda-time-2.9.9.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.tdunning/t-digest/3.2/2ab94758b0276a8a26102adf8d528cf6d0567b9a/t-digest-3.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.hdrhistogram/HdrHistogram/2.1.9/e4631ce165eb400edecfa32e03d3f1be53dee754/HdrHistogram-2.1.9.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.locationtech.spatial4j/spatial4j/0.7/faa8ba85d503da4ab872d17ba8c00da0098ab2f2/spatial4j-0.7.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.locationtech.jts/jts-core/1.15.0/705981b7e25d05a76a3654e597dab6ba423eb79e/jts-core-1.15.0.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.9.1/c041978c686866ee8534f538c6220238db3bb6be/log4j-core-2.9.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-1.2-api/2.9.1/894f96d677880d4ab834a1356f62b875e579caaa/log4j-1.2-api-2.9.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.elasticsearch/jna/4.5.1/da10908ae23dc59b19dc258e63aea1c44621dc3a/jna-4.5.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/5.0.2/98cafc6081d5632b61be2c9e60650b64ddbc637c/jopt-simple-5.0.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.yaml/snakeyaml/1.17/7a27ea250c5130b2922b86dea63cbb1cc10a660c/snakeyaml-1.17.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.8.10/eb21a035c66ad307e66ec8fce37f5d50fd62d039/jackson-core-2.8.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-smile/2.8.10/e853081fadaad3e98ed801937acc3d8f77580686/jackson-dataformat-smile-2.8.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-yaml/2.8.10/1e08caf1d787c825307d8cc6362452086020d853/jackson-dataformat-yaml-2.8.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-cbor/2.8.10/1c58cc9313ddf19f0900cd61ed044874278ce320/jackson-dataformat-cbor-2.8.10.jar] java.home: C:\Users\jenkins\.java\java8\jre examining directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\classes\java\test examining directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\resources\test examining directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\classes\java\main examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar no jar hell found :libs:elasticsearch-core:jarHell (Thread[Daemon worker,5,main]) completed. Took 0.362 secs. :libs:elasticsearch-core:licenseHeaders (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:licenseHeaders Task ':libs:elasticsearch-core:licenseHeaders' is not up-to-date because: No history is available. :libs:elasticsearch-core:licenseHeaders (Thread[Daemon worker,5,main]) completed. Took 0.057 secs. :libs:elasticsearch-core:loggerUsageCheck (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:loggerUsageCheck Task ':libs:elasticsearch-core:loggerUsageCheck' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core Command: C:\Users\jenkins\.java\java8\bin\java -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\distributions\logger-usage-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.ow2.asm\asm-debug-all\5.0.4\702b8525fcf81454235e5e2fa2a35f15ffc0ec7e\asm-debug-all-5.0.4.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar org.elasticsearch.test.loggerusage.ESLoggerUsageChecker C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\classes\java\main C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\classes\java\test Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' checking for wrong usages of ESLogger... No wrong usages found :libs:elasticsearch-core:loggerUsageCheck (Thread[Daemon worker,5,main]) completed. Took 0.305 secs. :libs:elasticsearch-core:namingConventions (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:namingConventions Task ':libs:elasticsearch-core:namingConventions' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core Command: C:\Users\jenkins\.java\java8\bin\java -Djna.nosys=true -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\distributions\build-tools-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\classes\java\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\resources\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\resources\test org.elasticsearch.test.NamingConventionsCheck --test-class org.apache.lucene.util.LuceneTestCase --integ-test-class org.elasticsearch.test.ESIntegTestCase -- C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\classes\java\test Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' :libs:elasticsearch-core:namingConventions (Thread[Daemon worker,5,main]) completed. Took 0.188 secs. :libs:elasticsearch-core:thirdPartyAudit (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:thirdPartyAudit SKIPPED Skipping task ':libs:elasticsearch-core:thirdPartyAudit' as task onlyIf is false. :libs:elasticsearch-core:thirdPartyAudit (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:elasticsearch-core:precommit (Thread[Daemon worker,5,main]) started. > Task :libs:elasticsearch-core:precommit Skipping task ':libs:elasticsearch-core:precommit' as it has no actions. :libs:elasticsearch-core:precommit (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:framework:copyCheckstyleConf (Thread[Daemon worker,5,main]) started. > Task :test:framework:copyCheckstyleConf Task ':test:framework:copyCheckstyleConf' is not up-to-date because: No history is available. :test:framework:copyCheckstyleConf (Thread[Daemon worker,5,main]) completed. Took 0.003 secs. :test:framework:checkstyleMain (Thread[Daemon worker,5,main]) started. > Task :test:framework:checkstyleMain Task ':test:framework:checkstyleMain' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 183 files :test:framework:checkstyleMain (Thread[Daemon worker,5,main]) completed. Took 1.891 secs. :test:framework:compileTestJava (Thread[Daemon worker,5,main]) started. > Task :test:framework:compileTestJava Task ':test:framework:compileTestJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':test:framework:compileTestJava'. :test:framework:compileTestJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\tmp\compileTestJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.047 secs. Created jar classpath snapshot for incremental compilation in 0.008 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :test:framework:compileTestJava (Thread[Daemon worker,5,main]) completed. Took 4.045 secs. :test:framework:processTestResources (Thread[Daemon worker,5,main]) started. > Task :test:framework:processTestResources Task ':test:framework:processTestResources' is not up-to-date because: No history is available. :test:framework:processTestResources (Thread[Daemon worker,5,main]) completed. Took 0.011 secs. :test:framework:testClasses (Thread[Daemon worker,5,main]) started. > Task :test:framework:testClasses Skipping task ':test:framework:testClasses' as it has no actions. :test:framework:testClasses (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:framework:checkstyleTest (Thread[Daemon worker,5,main]) started. > Task :test:framework:checkstyleTest Task ':test:framework:checkstyleTest' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 33 files :test:framework:checkstyleTest (Thread[Daemon worker,5,main]) completed. Took 0.512 secs. :test:framework:checkstyle (Thread[Daemon worker,5,main]) started. > Task :test:framework:checkstyle Skipping task ':test:framework:checkstyle' as it has no actions. :test:framework:checkstyle (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:framework:dependencyLicenses (Thread[Daemon worker,5,main]) started. > Task :test:framework:dependencyLicenses SKIPPED Skipping task ':test:framework:dependencyLicenses' as task onlyIf is false. :test:framework:dependencyLicenses (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:framework:filepermissions (Thread[Daemon worker,5,main]) started. > Task :test:framework:filepermissions SKIPPED Skipping task ':test:framework:filepermissions' as task onlyIf is false. :test:framework:filepermissions (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:framework:forbiddenApisMain (Thread[Daemon worker,5,main]) started. > Task :test:framework:forbiddenApisMain SKIPPED Skipping task ':test:framework:forbiddenApisMain' as task onlyIf is false. :test:framework:forbiddenApisMain (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:framework:forbiddenApisTest (Thread[Daemon worker,5,main]) started. > Task :test:framework:forbiddenApisTest Task ':test:framework:forbiddenApisTest' is not up-to-date because: No history is available. Reading bundled API signatures: jdk-unsafe-1.8 Reading bundled API signatures: jdk-deprecated-1.8 Reading bundled API signatures: jdk-non-portable Reading bundled API signatures: jdk-system-out Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/jdk-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-all-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-test-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/http-signatures.txt Loading classes to check... Scanning classes for violations... Scanned 60 class file(s) for forbidden API invocations (in 0.15s), 0 error(s). :test:framework:forbiddenApisTest (Thread[Daemon worker,5,main]) completed. Took 0.172 secs. :test:framework:forbiddenApis (Thread[Daemon worker,5,main]) started. > Task :test:framework:forbiddenApis Skipping task ':test:framework:forbiddenApis' as it has no actions. :test:framework:forbiddenApis (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:framework:forbiddenPatterns (Thread[Daemon worker,5,main]) started. > Task :test:framework:forbiddenPatterns Task ':test:framework:forbiddenPatterns' is not up-to-date because: No history is available. :test:framework:forbiddenPatterns (Thread[Daemon worker,5,main]) completed. Took 0.149 secs. :test:framework:jarHell (Thread[Daemon worker,5,main]) started. > Task :test:framework:jarHell SKIPPED Skipping task ':test:framework:jarHell' as task onlyIf is false. :test:framework:jarHell (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:framework:licenseHeaders (Thread[Daemon worker,5,main]) started. > Task :test:framework:licenseHeaders Task ':test:framework:licenseHeaders' is not up-to-date because: No history is available. :test:framework:licenseHeaders (Thread[Daemon worker,5,main]) completed. Took 0.105 secs. :test:framework:loggerUsageCheck (Thread[Daemon worker,5,main]) started. > Task :test:framework:loggerUsageCheck Task ':test:framework:loggerUsageCheck' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework Command: C:\Users\jenkins\.java\java8\bin\java -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\distributions\logger-usage-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.ow2.asm\asm-debug-all\5.0.4\702b8525fcf81454235e5e2fa2a35f15ffc0ec7e\asm-debug-all-5.0.4.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar org.elasticsearch.test.loggerusage.ESLoggerUsageChecker C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\classes\java\main C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\classes\java\test Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' checking for wrong usages of ESLogger... No wrong usages found :test:framework:loggerUsageCheck (Thread[Daemon worker,5,main]) completed. Took 0.873 secs. :test:framework:namingConventions (Thread[Daemon worker,5,main]) started. > Task :test:framework:namingConventions Task ':test:framework:namingConventions' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework Command: C:\Users\jenkins\.java\java8\bin\java -Djna.nosys=true -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\distributions\build-tools-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\classes\java\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\resources\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\resources\test org.elasticsearch.test.NamingConventionsCheck --test-class org.apache.lucene.util.LuceneTestCase --integ-test-class org.elasticsearch.test.ESIntegTestCase -- C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\classes\java\test Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' :test:framework:namingConventions (Thread[Daemon worker,5,main]) completed. Took 0.311 secs. :test:framework:namingConventionsMain (Thread[Daemon worker,5,main]) started. > Task :test:framework:namingConventionsMain Task ':test:framework:namingConventionsMain' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework Command: C:\Users\jenkins\.java\java8\bin\java -Djna.nosys=true -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\distributions\build-tools-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\classes\java\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\resources\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\resources\test org.elasticsearch.test.NamingConventionsCheck --test-class org.apache.lucene.util.LuceneTestCase --integ-test-class org.elasticsearch.test.ESIntegTestCase --main -- C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\classes\java\main Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' :test:framework:namingConventionsMain (Thread[Daemon worker,5,main]) completed. Took 0.546 secs. :test:framework:thirdPartyAudit (Thread[Daemon worker,5,main]) started. > Task :test:framework:thirdPartyAudit Task ':test:framework:thirdPartyAudit' is not up-to-date because: No history is available. [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\tmp\thirdPartyAudit [thirdPartyAudit] Reading inline API signatures... [thirdPartyAudit] Loading classes to check... [thirdPartyAudit] Scanning classes for violations... [thirdPartyAudit] Scanned 1520 class file(s) for forbidden API invocations (in 1.69s), 0 error(s). [delete] Deleting directory C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\tmp\thirdPartyAudit :test:framework:thirdPartyAudit (Thread[Daemon worker,5,main]) completed. Took 3.04 secs. :test:framework:precommit (Thread[Daemon worker,5,main]) started. > Task :test:framework:precommit Skipping task ':test:framework:precommit' as it has no actions. :test:framework:precommit (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:framework:test (Thread[Daemon worker,5,main]) started. Started J2 PID(7248@SLAVE-0633DD87F). Started J1 PID(9704@SLAVE-0633DD87F). Started J0 PID(11432@SLAVE-0633DD87F). > Task :test:framework:test Task ':test:framework:test' is not up-to-date because: Task.upToDateWhen is false. [ant:junit4] says ??! Master seed: 6A48B651990D67B ==> Test Info: seed=6A48B651990D67B; jvms=3; suites=28 Suite: org.elasticsearch.test.test.ESTestCaseTests Completed [1/28] on J2 in 5.27s, 7 tests Suite: org.elasticsearch.test.AbstractQueryTestCaseTests Completed [2/28] on J2 in 0.17s, 2 tests Suite: org.elasticsearch.test.rest.yaml.section.TeardownSectionTests Completed [3/28] on J1 in 5.74s, 2 tests Suite: org.elasticsearch.test.rest.yaml.section.DoSectionTests IGNOR/A 0.02s J0 | DoSectionTests.testParseDoSectionWithJsonMultipleBodiesRepeatedProperty > Assumption #1: Test only makes sense if XContent parser doesn't have strict duplicate checks enabled IGNOR/A 0.01s J0 | DoSectionTests.testParseDoSectionWithYamlMultipleBodiesRepeatedProperty > Assumption #1: Test only makes sense if XContent parser doesn't have strict duplicate checks enabled Completed [4/28] on J0 in 5.99s, 17 tests, 2 skipped Suite: org.elasticsearch.test.disruption.NetworkDisruptionTests Completed [5/28] on J0 in 0.22s, 5 tests Suite: org.elasticsearch.test.rest.yaml.StashTests Completed [6/28] on J0 in 0.12s, 6 tests Suite: org.elasticsearch.test.rest.yaml.restspec.ClientYamlSuiteRestApiParserTests Completed [7/28] on J2 in 1.07s, 3 tests Suite: org.elasticsearch.test.rest.yaml.section.AssertionTests Completed [8/28] on J0 in 0.20s, 9 tests Suite: org.elasticsearch.test.rest.yaml.section.ClientYamlTestSuiteTests Completed [9/28] on J2 in 0.71s, 4 tests Suite: org.elasticsearch.test.rest.yaml.ObjectPathTests Completed [10/28] on J2 in 0.28s, 9 tests Suite: org.elasticsearch.test.rest.yaml.section.SetupSectionTests Completed [11/28] on J2 in 0.06s, 2 tests Suite: org.elasticsearch.test.rest.yaml.BlacklistedPathPatternMatcherTests Completed [12/28] on J2 in 0.05s, 5 tests Suite: org.elasticsearch.search.MockSearchServiceTests Completed [13/28] on J1 in 2.00s, 1 test Suite: org.elasticsearch.test.rest.yaml.restspec.ClientYamlSuiteRestApiParserFailingTests IGNOR/A 0.05s J1 | ClientYamlSuiteRestApiParserFailingTests.testDuplicateParts > Assumption #1: Test only makes sense if XContent parser doesn't have strict duplicate checks enabled IGNOR/A 0.00s J1 | ClientYamlSuiteRestApiParserFailingTests.testDuplicateParams > Assumption #1: Test only makes sense if XContent parser doesn't have strict duplicate checks enabled Completed [14/28] on J1 in 0.24s, 7 tests, 2 skipped Suite: org.elasticsearch.test.rest.yaml.section.ClientYamlTestSectionTests Completed [15/28] on J1 in 0.68s, 9 tests Suite: org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCaseTests Completed [16/28] on J1 in 0.14s, 3 tests Suite: org.elasticsearch.test.rest.yaml.section.SetSectionTests Completed [17/28] on J1 in 0.04s, 3 tests Suite: org.elasticsearch.test.hamcrest.ElasticsearchAssertionsTests Completed [18/28] on J1 in 0.12s, 2 tests Suite: org.elasticsearch.test.VersionUtilsTests Completed [19/28] on J1 in 0.29s, 10 tests Suite: org.elasticsearch.test.XContentTestUtilsTests Completed [20/28] on J1 in 0.12s, 3 tests Suite: org.elasticsearch.test.disruption.LongGCDisruptionTests Completed [21/28] on J1 in 3.25s, 3 tests Suite: org.elasticsearch.test.test.LoggingListenerTests Completed [22/28] on J1 in 0.07s, 6 tests HEARTBEAT J0 PID(11432@SLAVE-0633DD87F): 2018-05-05T08:25:51, stalled for 10.2s at: InternalTestClusterTests.testDifferentRolesMaintainPathOnRestart HEARTBEAT J2 PID(7248@SLAVE-0633DD87F): 2018-05-05T08:25:53, stalled for 11.0s at: MockNodeTests.testComponentsMockedByMarkerPlugins Suite: org.elasticsearch.node.MockNodeTests Completed [23/28] on J2 in 11.86s, 1 test Suite: org.elasticsearch.transport.MockTcpTransportTests Completed [24/28] on J1 in 10.38s, 39 tests HEARTBEAT J0 PID(11432@SLAVE-0633DD87F): 2018-05-05T08:26:11, stalled for 11.9s at: InternalTestClusterTests.testBeforeTest HEARTBEAT J0 PID(11432@SLAVE-0633DD87F): 2018-05-05T08:26:21, stalled for 21.9s at: InternalTestClusterTests.testBeforeTest HEARTBEAT J0 PID(11432@SLAVE-0633DD87F): 2018-05-05T08:26:31, stalled for 31.9s at: InternalTestClusterTests.testBeforeTest Suite: org.elasticsearch.test.test.InternalTestClusterTests Completed [25/28] on J0 in 58.38s, 5 tests Suite: org.elasticsearch.test.rest.yaml.ClientYamlTestExecutionContextTests Completed [26/28] on J0 in 0.05s, 1 test Suite: org.elasticsearch.transport.nio.SimpleMockNioTransportTests Completed [27/28] on J0 in 7.43s, 41 tests Suite: org.elasticsearch.test.rest.yaml.section.SkipSectionTests Completed [28/28] on J0 in 0.04s, 9 tests > Task :test:framework:test Slow Tests Summary: 58.38s | org.elasticsearch.test.test.InternalTestClusterTests 11.86s | org.elasticsearch.node.MockNodeTests 10.38s | org.elasticsearch.transport.MockTcpTransportTests 7.43s | org.elasticsearch.transport.nio.SimpleMockNioTransportTests 5.99s | org.elasticsearch.test.rest.yaml.section.DoSectionTests ==> Test Summary: 28 suites, 214 tests, 4 ignored (4 assumptions) [ant:junit4] JVM J0: 0.95 .. 74.10 = 73.15s [ant:junit4] JVM J1: 0.95 .. 25.65 = 24.70s [ant:junit4] JVM J2: 0.46 .. 21.04 = 20.58s [ant:junit4] Execution time total: 1 minute 14 seconds [ant:junit4] Tests summary: 28 suites, 214 tests, 4 ignored (4 assumptions) :test:framework:test (Thread[Daemon worker,5,main]) completed. Took 1 mins 14.25 secs. :test:logger-usage:copyCheckstyleConf (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:copyCheckstyleConf Task ':test:logger-usage:copyCheckstyleConf' is not up-to-date because: No history is available. :test:logger-usage:copyCheckstyleConf (Thread[Daemon worker,5,main]) completed. Took 0.005 secs. :test:logger-usage:checkstyleMain (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:checkstyleMain Task ':test:logger-usage:checkstyleMain' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 2 files :test:logger-usage:checkstyleMain (Thread[Daemon worker,5,main]) completed. Took 0.155 secs. :test:logger-usage:compileTestJava (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:compileTestJava Task ':test:logger-usage:compileTestJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':test:logger-usage:compileTestJava'. :test:logger-usage:compileTestJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\tmp\compileTestJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.003 secs. Created jar classpath snapshot for incremental compilation in 0.009 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :test:logger-usage:compileTestJava (Thread[Daemon worker,5,main]) completed. Took 1.559 secs. :test:logger-usage:processTestResources (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:processTestResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\src\test\resources', not found Skipping task ':test:logger-usage:processTestResources' as it has no source files and no previous output files. :test:logger-usage:processTestResources (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:logger-usage:testClasses (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:testClasses Skipping task ':test:logger-usage:testClasses' as it has no actions. :test:logger-usage:testClasses (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:logger-usage:checkstyleTest (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:checkstyleTest Task ':test:logger-usage:checkstyleTest' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 1 files :test:logger-usage:checkstyleTest (Thread[Daemon worker,5,main]) completed. Took 0.099 secs. :test:logger-usage:checkstyle (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:checkstyle Skipping task ':test:logger-usage:checkstyle' as it has no actions. :test:logger-usage:checkstyle (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:logger-usage:dependencyLicenses (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:dependencyLicenses SKIPPED Skipping task ':test:logger-usage:dependencyLicenses' as task onlyIf is false. :test:logger-usage:dependencyLicenses (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:logger-usage:filepermissions (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:filepermissions SKIPPED Skipping task ':test:logger-usage:filepermissions' as task onlyIf is false. :test:logger-usage:filepermissions (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:logger-usage:forbiddenApisMain (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:forbiddenApisMain Task ':test:logger-usage:forbiddenApisMain' is not up-to-date because: No history is available. Reading bundled API signatures: jdk-unsafe-1.8 Reading bundled API signatures: jdk-deprecated-1.8 Reading bundled API signatures: jdk-non-portable Reading bundled API signatures: jdk-system-out Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/jdk-signatures.txt Loading classes to check... Scanning classes for violations... Scanned 12 class file(s) for forbidden API invocations (in 0.03s), 0 error(s). :test:logger-usage:forbiddenApisMain (Thread[Daemon worker,5,main]) completed. Took 0.036 secs. :test:logger-usage:forbiddenApisTest (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:forbiddenApisTest Task ':test:logger-usage:forbiddenApisTest' is not up-to-date because: No history is available. Reading bundled API signatures: jdk-unsafe-1.8 Reading bundled API signatures: jdk-deprecated-1.8 Reading bundled API signatures: jdk-non-portable Reading bundled API signatures: jdk-system-out Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/jdk-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-all-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-test-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/http-signatures.txt Loading classes to check... Scanning classes for violations... Scanned 1 class file(s) for forbidden API invocations (in 0.04s), 0 error(s). :test:logger-usage:forbiddenApisTest (Thread[Daemon worker,5,main]) completed. Took 0.049 secs. :test:logger-usage:forbiddenApis (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:forbiddenApis Skipping task ':test:logger-usage:forbiddenApis' as it has no actions. :test:logger-usage:forbiddenApis (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:logger-usage:forbiddenPatterns (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:forbiddenPatterns Task ':test:logger-usage:forbiddenPatterns' is not up-to-date because: No history is available. file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\src\main\resources', not found file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\src\test\resources', not found :test:logger-usage:forbiddenPatterns (Thread[Daemon worker,5,main]) completed. Took 0.005 secs. :test:logger-usage:jarHell (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:jarHell Task ':test:logger-usage:jarHell' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage Command: C:\Users\jenkins\.java\java8\bin\java -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\classes\java\main;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.ow2.asm\asm-debug-all\5.0.4\702b8525fcf81454235e5e2fa2a35f15ffc0ec7e\asm-debug-all-5.0.4.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar org.elasticsearch.bootstrap.JarHell Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' checking for jar hell... java.class.path: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\classes\java\main;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.ow2.asm\asm-debug-all\5.0.4\702b8525fcf81454235e5e2fa2a35f15ffc0ec7e\asm-debug-all-5.0.4.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar sun.boot.class.path: C:\Users\jenkins\.java\java8\jre\lib\resources.jar;C:\Users\jenkins\.java\java8\jre\lib\rt.jar;C:\Users\jenkins\.java\java8\jre\lib\sunrsasign.jar;C:\Users\jenkins\.java\java8\jre\lib\jsse.jar;C:\Users\jenkins\.java\java8\jre\lib\jce.jar;C:\Users\jenkins\.java\java8\jre\lib\charsets.jar;C:\Users\jenkins\.java\java8\jre\lib\jfr.jar;C:\Users\jenkins\.java\java8\jre\classes classloader urls: [file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/test/logger-usage/build/classes/java/test/, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/test/logger-usage/build/classes/java/main/, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.ow2.asm/asm-debug-all/5.0.4/702b8525fcf81454235e5e2fa2a35f15ffc0ec7e/asm-debug-all-5.0.4.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/test/framework/build/distributions/framework-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/elasticsearch-nio/build/distributions/elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/server/build/distributions/elasticsearch-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.9.1/7a2999229464e7a324aa503c0a52ec0f05efe7bd/log4j-api-2.9.1.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/client/rest/build/distributions/elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/server/cli/build/distributions/elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.carrotsearch.randomizedtesting/randomizedtesting-runner/2.5.2/91f3284993b44dcb2f003b5f28617abba13971d2/randomizedtesting-runner-2.5.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/junit/junit/4.12/2973d150c0dc1fefe998f834810d68f278ea58ec/junit-4.12.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-all/1.3/63a21ebc981131004ad02e0434e799fd7f3a8d5a/hamcrest-all-1.3.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-test-framework/7.4.0-snapshot-1ed95c097b/1afd168832ee0cf36d1adcc960c3e127d8848bd0/lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-codecs/7.4.0-snapshot-1ed95c097b/4ad08cbda4541cbf5899860aa99c08e557585f74/lucene-codecs-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/commons-logging/commons-logging/1.1.3/f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f/commons-logging-1.1.3.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/commons-codec/commons-codec/1.10/4b95f4897fa13f2cd904aee711aeafc0c5295cd8/commons-codec-1.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.elasticsearch/securemock/1.2/98201d4ad5ac93f6b415ae9172d52b5e7cda490e/securemock-1.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.elasticsearch/mocksocket/1.2/190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361/mocksocket-1.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpclient/4.5.2/733db77aa8d9b2d68015189df76ab06304406e50/httpclient-4.5.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpcore/4.4.5/e7501a1b34325abb00d17dde96150604a0658b54/httpcore-4.4.5.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpasyncclient/4.1.2/95aa3e6fb520191a0970a73cf09f62948ee614be/httpasyncclient-4.1.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpcore-nio/4.4.5/f4be009e7505f6ceddf21e7960c759f413f15056/httpcore-nio-4.4.5.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/x-content/build/distributions/elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/elasticsearch-core/build/distributions/elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/secure-sm/build/distributions/elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/7.4.0-snapshot-1ed95c097b/b06e4f6514256a3f187a9892e520638b9c59e63/lucene-core-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/7.4.0-snapshot-1ed95c097b/2b2be48f6622c150496e755497e7bdb8daa46030/lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-backward-codecs/7.4.0-snapshot-1ed95c097b/6cbafc48e8ac4966377665eb3bbe93f9addf04a5/lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-grouping/7.4.0-snapshot-1ed95c097b/4c71cef87fe513a7a96c2a7980ed6f7c2b015763/lucene-grouping-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-highlighter/7.4.0-snapshot-1ed95c097b/665e044d1180100940bccd7e8e41dde48e342da3/lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-join/7.4.0-snapshot-1ed95c097b/d343bbf5792f5969288b59b51179acd29d04f4ee/lucene-join-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-memory/7.4.0-snapshot-1ed95c097b/8915f3c93af3348655bcc204289f9011835738a2/lucene-memory-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-misc/7.4.0-snapshot-1ed95c097b/e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d/lucene-misc-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queries/7.4.0-snapshot-1ed95c097b/5946d5e2be276f66e9ff6d6111acabb03a9330d9/lucene-queries-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/7.4.0-snapshot-1ed95c097b/d9fc5fc63f3d861e5af72e11373368e8a4c6bba6/lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/7.4.0-snapshot-1ed95c097b/ee283c0a1a717f3e0915de75864a93d043efaee3/lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial/7.4.0-snapshot-1ed95c097b/e1adf0220a7c052ac81e2919ffac24ac0e5b007c/lucene-spatial-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial-extras/7.4.0-snapshot-1ed95c097b/6d9306053942c48f43392a634f11a95462b5996e/lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial3d/7.4.0-snapshot-1ed95c097b/2334e8c5f4d0f98659b30e0c2035296e4aae8ff5/lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-suggest/7.4.0-snapshot-1ed95c097b/f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463/lucene-suggest-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.carrotsearch/hppc/0.7.1/8b5057f74ea378c0150a1860874a3ebdcb713767/hppc-0.7.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/joda-time/joda-time/2.9.9/f7b520c458572890807d143670c9b24f4de90897/joda-time-2.9.9.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.tdunning/t-digest/3.2/2ab94758b0276a8a26102adf8d528cf6d0567b9a/t-digest-3.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.hdrhistogram/HdrHistogram/2.1.9/e4631ce165eb400edecfa32e03d3f1be53dee754/HdrHistogram-2.1.9.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.locationtech.spatial4j/spatial4j/0.7/faa8ba85d503da4ab872d17ba8c00da0098ab2f2/spatial4j-0.7.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.locationtech.jts/jts-core/1.15.0/705981b7e25d05a76a3654e597dab6ba423eb79e/jts-core-1.15.0.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.9.1/c041978c686866ee8534f538c6220238db3bb6be/log4j-core-2.9.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-1.2-api/2.9.1/894f96d677880d4ab834a1356f62b875e579caaa/log4j-1.2-api-2.9.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.elasticsearch/jna/4.5.1/da10908ae23dc59b19dc258e63aea1c44621dc3a/jna-4.5.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/5.0.2/98cafc6081d5632b61be2c9e60650b64ddbc637c/jopt-simple-5.0.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.yaml/snakeyaml/1.17/7a27ea250c5130b2922b86dea63cbb1cc10a660c/snakeyaml-1.17.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.8.10/eb21a035c66ad307e66ec8fce37f5d50fd62d039/jackson-core-2.8.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-smile/2.8.10/e853081fadaad3e98ed801937acc3d8f77580686/jackson-dataformat-smile-2.8.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-yaml/2.8.10/1e08caf1d787c825307d8cc6362452086020d853/jackson-dataformat-yaml-2.8.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-cbor/2.8.10/1c58cc9313ddf19f0900cd61ed044874278ce320/jackson-dataformat-cbor-2.8.10.jar] java.home: C:\Users\jenkins\.java\java8\jre examining directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\classes\java\test examining directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\classes\java\main examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.ow2.asm\asm-debug-all\5.0.4\702b8525fcf81454235e5e2fa2a35f15ffc0ec7e\asm-debug-all-5.0.4.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar no jar hell found :test:logger-usage:jarHell (Thread[Daemon worker,5,main]) completed. Took 0.366 secs. :test:logger-usage:licenseHeaders (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:licenseHeaders Task ':test:logger-usage:licenseHeaders' is not up-to-date because: No history is available. :test:logger-usage:licenseHeaders (Thread[Daemon worker,5,main]) completed. Took 0.058 secs. :test:logger-usage:loggerUsageCheck (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:loggerUsageCheck SKIPPED Skipping task ':test:logger-usage:loggerUsageCheck' as task onlyIf is false. :test:logger-usage:loggerUsageCheck (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:logger-usage:namingConventions (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:namingConventions Task ':test:logger-usage:namingConventions' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage Command: C:\Users\jenkins\.java\java8\bin\java -Djna.nosys=true -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\distributions\build-tools-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\classes\java\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\resources\main;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.ow2.asm\asm-debug-all\5.0.4\702b8525fcf81454235e5e2fa2a35f15ffc0ec7e\asm-debug-all-5.0.4.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\resources\test org.elasticsearch.test.NamingConventionsCheck --test-class org.apache.lucene.util.LuceneTestCase --integ-test-class org.elasticsearch.test.ESIntegTestCase -- C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\classes\java\test Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' :test:logger-usage:namingConventions (Thread[Daemon worker,5,main]) completed. Took 0.213 secs. :test:logger-usage:thirdPartyAudit (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:thirdPartyAudit Task ':test:logger-usage:thirdPartyAudit' is not up-to-date because: No history is available. [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.ow2.asm\asm-debug-all\5.0.4\702b8525fcf81454235e5e2fa2a35f15ffc0ec7e\asm-debug-all-5.0.4.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\tmp\thirdPartyAudit [thirdPartyAudit] Reading inline API signatures... [thirdPartyAudit] Loading classes to check... [thirdPartyAudit] Scanning classes for violations... [thirdPartyAudit] Scanned 311 class file(s) for forbidden API invocations (in 0.44s), 0 error(s). [delete] Deleting directory C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\tmp\thirdPartyAudit :test:logger-usage:thirdPartyAudit (Thread[Daemon worker,5,main]) completed. Took 0.779 secs. :test:logger-usage:precommit (Thread[Daemon worker,5,main]) started. > Task :test:logger-usage:precommit Skipping task ':test:logger-usage:precommit' as it has no actions. :test:logger-usage:precommit (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :test:logger-usage:test (Thread[Daemon worker,5,main]) started. Started J0 PID(11232@SLAVE-0633DD87F). > Task :test:logger-usage:test Task ':test:logger-usage:test' is not up-to-date because: Task.upToDateWhen is false. [ant:junit4] says ??! Master seed: ADF5DEFE57AAE849 ==> Test Info: seed=ADF5DEFE57AAE849; jvm=1; suite=1 Suite: org.elasticsearch.test.loggerusage.ESLoggerUsageTests Completed [1/1] in 1.94s, 2 tests > Task :test:logger-usage:test Slow Tests Summary: 1.94s | org.elasticsearch.test.loggerusage.ESLoggerUsageTests ==> Test Summary: 1 suite, 2 tests [ant:junit4] JVM J0: 0.27 .. 2.79 = 2.51s [ant:junit4] Execution time total: 2.79 sec. [ant:junit4] Tests summary: 1 suite, 2 tests :test:logger-usage:test (Thread[Daemon worker,5,main]) completed. Took 2.812 secs. :libs:elasticsearch-core:test (Thread[Daemon worker,5,main]) started. Started J1 PID(8104@SLAVE-0633DD87F). Started J2 PID(5736@SLAVE-0633DD87F). Started J0 PID(796@SLAVE-0633DD87F). > Task :libs:elasticsearch-core:test Task ':libs:elasticsearch-core:test' is not up-to-date because: Task.upToDateWhen is false. [ant:junit4] says hallo! Master seed: 92281325CAA84FFD ==> Test Info: seed=92281325CAA84FFD; jvms=3; suites=5 Suite: org.elasticsearch.common.unit.TimeValueTests Completed [1/5] on J2 in 4.25s, 15 tests Suite: org.elasticsearch.common.collect.TupleTests Completed [2/5] on J1 in 4.35s, 1 test Suite: org.elasticsearch.bootstrap.JarHellTests IGNOR/A 0.02s J0 | JarHellTests.testEmptyClassPathUnix > Assumption #1: test is designed for unix-like systems only IGNOR/A 0.00s J0 | JarHellTests.testParseClassPathUnix > Assumption #1: test is designed for unix-like systems only Completed [3/5] on J0 in 4.57s, 15 tests, 2 skipped Suite: org.elasticsearch.core.internal.io.StreamsTests Completed [4/5] on J1 in 0.15s, 1 test Suite: org.elasticsearch.core.internal.io.IOUtilsTests Completed [5/5] on J0 in 0.69s, 10 tests > Task :libs:elasticsearch-core:test Slow Tests Summary: 4.57s | org.elasticsearch.bootstrap.JarHellTests 4.35s | org.elasticsearch.common.collect.TupleTests 4.25s | org.elasticsearch.common.unit.TimeValueTests 0.69s | org.elasticsearch.core.internal.io.IOUtilsTests 0.15s | org.elasticsearch.core.internal.io.StreamsTests ==> Test Summary: 5 suites, 42 tests, 2 ignored (2 assumptions) [ant:junit4] JVM J0: 0.57 .. 6.69 = 6.12s [ant:junit4] JVM J1: 0.57 .. 5.71 = 5.14s [ant:junit4] JVM J2: 0.57 .. 5.66 = 5.09s [ant:junit4] Execution time total: 6.70 sec. [ant:junit4] Tests summary: 5 suites, 42 tests, 2 ignored (2 assumptions) :libs:elasticsearch-core:test (Thread[Daemon worker,5,main]) completed. Took 6.723 secs. :libs:plugin-classloader:copyCheckstyleConf (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:copyCheckstyleConf Task ':libs:plugin-classloader:copyCheckstyleConf' is not up-to-date because: No history is available. :libs:plugin-classloader:copyCheckstyleConf (Thread[Daemon worker,5,main]) completed. Took 0.004 secs. :libs:plugin-classloader:checkstyleMain (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:checkstyleMain Task ':libs:plugin-classloader:checkstyleMain' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 1 files :libs:plugin-classloader:checkstyleMain (Thread[Daemon worker,5,main]) completed. Took 0.061 secs. :libs:plugin-classloader:compileTestJava (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:compileTestJava NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader\src\test\java', not found Skipping task ':libs:plugin-classloader:compileTestJava' as it has no source files and no previous output files. :libs:plugin-classloader:compileTestJava (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:plugin-classloader:processTestResources (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:processTestResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader\src\test\resources', not found Skipping task ':libs:plugin-classloader:processTestResources' as it has no source files and no previous output files. :libs:plugin-classloader:processTestResources (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:plugin-classloader:testClasses (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:testClasses UP-TO-DATE Skipping task ':libs:plugin-classloader:testClasses' as it has no actions. :libs:plugin-classloader:testClasses (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:plugin-classloader:checkstyleTest (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:checkstyleTest NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader\src\test\java', not found Skipping task ':libs:plugin-classloader:checkstyleTest' as it has no source files and no previous output files. :libs:plugin-classloader:checkstyleTest (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:plugin-classloader:checkstyle (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:checkstyle Skipping task ':libs:plugin-classloader:checkstyle' as it has no actions. :libs:plugin-classloader:checkstyle (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:plugin-classloader:dependencyLicenses (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:dependencyLicenses Task ':libs:plugin-classloader:dependencyLicenses' is not up-to-date because: Task has not declared any outputs despite executing actions. :libs:plugin-classloader:dependencyLicenses (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:plugin-classloader:filepermissions (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:filepermissions SKIPPED Skipping task ':libs:plugin-classloader:filepermissions' as task onlyIf is false. :libs:plugin-classloader:filepermissions (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:plugin-classloader:forbiddenApisMain (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:forbiddenApisMain SKIPPED Skipping task ':libs:plugin-classloader:forbiddenApisMain' as task onlyIf is false. :libs:plugin-classloader:forbiddenApisMain (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:plugin-classloader:forbiddenApisTest (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:forbiddenApisTest NO-SOURCE Skipping task ':libs:plugin-classloader:forbiddenApisTest' as it has no source files and no previous output files. :libs:plugin-classloader:forbiddenApisTest (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:plugin-classloader:forbiddenApis (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:forbiddenApis UP-TO-DATE Skipping task ':libs:plugin-classloader:forbiddenApis' as it has no actions. :libs:plugin-classloader:forbiddenApis (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:plugin-classloader:forbiddenPatterns (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:forbiddenPatterns Task ':libs:plugin-classloader:forbiddenPatterns' is not up-to-date because: No history is available. file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader\src\main\resources', not found file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader\src\test\resources', not found file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader\src\test\java', not found :libs:plugin-classloader:forbiddenPatterns (Thread[Daemon worker,5,main]) completed. Took 0.003 secs. :libs:plugin-classloader:jarHell (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:jarHell SKIPPED Skipping task ':libs:plugin-classloader:jarHell' as task onlyIf is false. :libs:plugin-classloader:jarHell (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:plugin-classloader:licenseHeaders (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:licenseHeaders file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader\src\test\java', not found Task ':libs:plugin-classloader:licenseHeaders' is not up-to-date because: No history is available. :libs:plugin-classloader:licenseHeaders (Thread[Daemon worker,5,main]) completed. Took 0.079 secs. :libs:plugin-classloader:loggerUsageCheck (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:loggerUsageCheck Task ':libs:plugin-classloader:loggerUsageCheck' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader Command: C:\Users\jenkins\.java\java8\bin\java -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\distributions\logger-usage-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.ow2.asm\asm-debug-all\5.0.4\702b8525fcf81454235e5e2fa2a35f15ffc0ec7e\asm-debug-all-5.0.4.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar org.elasticsearch.test.loggerusage.ESLoggerUsageChecker C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader\build\classes\java\main Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' checking for wrong usages of ESLogger... No wrong usages found :libs:plugin-classloader:loggerUsageCheck (Thread[Daemon worker,5,main]) completed. Took 0.264 secs. :libs:plugin-classloader:namingConventions (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:namingConventions SKIPPED Skipping task ':libs:plugin-classloader:namingConventions' as task onlyIf is false. :libs:plugin-classloader:namingConventions (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:plugin-classloader:thirdPartyAudit (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:thirdPartyAudit SKIPPED Skipping task ':libs:plugin-classloader:thirdPartyAudit' as task onlyIf is false. :libs:plugin-classloader:thirdPartyAudit (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:plugin-classloader:precommit (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:precommit Skipping task ':libs:plugin-classloader:precommit' as it has no actions. :libs:plugin-classloader:precommit (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:plugin-classloader:test (Thread[Daemon worker,5,main]) started. > Task :libs:plugin-classloader:test SKIPPED Skipping task ':libs:plugin-classloader:test' as task onlyIf is false. :libs:plugin-classloader:test (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:secure-sm:copyCheckstyleConf (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:copyCheckstyleConf Task ':libs:secure-sm:copyCheckstyleConf' is not up-to-date because: No history is available. :libs:secure-sm:copyCheckstyleConf (Thread[Daemon worker,5,main]) completed. Took 0.004 secs. :libs:secure-sm:checkstyleMain (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:checkstyleMain Task ':libs:secure-sm:checkstyleMain' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 3 files :libs:secure-sm:checkstyleMain (Thread[Daemon worker,5,main]) completed. Took 0.09 secs. :libs:secure-sm:compileTestJava (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:compileTestJava Task ':libs:secure-sm:compileTestJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':libs:secure-sm:compileTestJava'. :libs:secure-sm:compileTestJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\tmp\compileTestJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.005 secs. Created jar classpath snapshot for incremental compilation in 0.005 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :libs:secure-sm:compileTestJava (Thread[Daemon worker,5,main]) completed. Took 1.298 secs. :libs:secure-sm:processTestResources (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:processTestResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\src\test\resources', not found Skipping task ':libs:secure-sm:processTestResources' as it has no source files and no previous output files. :libs:secure-sm:processTestResources (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:secure-sm:testClasses (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:testClasses Skipping task ':libs:secure-sm:testClasses' as it has no actions. :libs:secure-sm:testClasses (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:secure-sm:checkstyleTest (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:checkstyleTest Task ':libs:secure-sm:checkstyleTest' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 2 files :libs:secure-sm:checkstyleTest (Thread[Daemon worker,5,main]) completed. Took 0.077 secs. :libs:secure-sm:checkstyle (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:checkstyle Skipping task ':libs:secure-sm:checkstyle' as it has no actions. :libs:secure-sm:checkstyle (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:secure-sm:dependencyLicenses (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:dependencyLicenses Task ':libs:secure-sm:dependencyLicenses' is not up-to-date because: Task has not declared any outputs despite executing actions. :libs:secure-sm:dependencyLicenses (Thread[Daemon worker,5,main]) completed. Took 0.001 secs. :libs:secure-sm:filepermissions (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:filepermissions SKIPPED Skipping task ':libs:secure-sm:filepermissions' as task onlyIf is false. :libs:secure-sm:filepermissions (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:secure-sm:forbiddenApisMain (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:forbiddenApisMain Task ':libs:secure-sm:forbiddenApisMain' is not up-to-date because: No history is available. Reading bundled API signatures: jdk-unsafe-1.8 Reading bundled API signatures: jdk-deprecated-1.8 Reading bundled API signatures: jdk-non-portable Reading bundled API signatures: jdk-system-out Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/jdk-signatures.txt Loading classes to check... Scanning classes for violations... Scanned 5 class file(s) for forbidden API invocations (in 0.02s), 0 error(s). :libs:secure-sm:forbiddenApisMain (Thread[Daemon worker,5,main]) completed. Took 0.029 secs. :libs:secure-sm:forbiddenApisTest (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:forbiddenApisTest Task ':libs:secure-sm:forbiddenApisTest' is not up-to-date because: No history is available. Reading bundled API signatures: jdk-unsafe-1.8 Reading bundled API signatures: jdk-deprecated-1.8 Reading bundled API signatures: jdk-non-portable Reading bundled API signatures: jdk-system-out Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/jdk-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-all-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-test-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/http-signatures.txt Loading classes to check... Scanning classes for violations... Scanned 6 class file(s) for forbidden API invocations (in 0.05s), 0 error(s). :libs:secure-sm:forbiddenApisTest (Thread[Daemon worker,5,main]) completed. Took 0.056 secs. :libs:secure-sm:forbiddenApis (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:forbiddenApis Skipping task ':libs:secure-sm:forbiddenApis' as it has no actions. :libs:secure-sm:forbiddenApis (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:secure-sm:forbiddenPatterns (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:forbiddenPatterns Task ':libs:secure-sm:forbiddenPatterns' is not up-to-date because: No history is available. file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\src\main\resources', not found file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\src\test\resources', not found :libs:secure-sm:forbiddenPatterns (Thread[Daemon worker,5,main]) completed. Took 0.006 secs. :libs:secure-sm:jarHell (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:jarHell SKIPPED Skipping task ':libs:secure-sm:jarHell' as task onlyIf is false. :libs:secure-sm:jarHell (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :libs:secure-sm:licenseHeaders (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:licenseHeaders Task ':libs:secure-sm:licenseHeaders' is not up-to-date because: No history is available. :libs:secure-sm:licenseHeaders (Thread[Daemon worker,5,main]) completed. Took 0.038 secs. :libs:secure-sm:loggerUsageCheck (Thread[Daemon worker,5,main]) started. > Task :libs:secure-sm:loggerUsageCheck Task ':libs:secure-sm:loggerUsageCheck' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm Command: C:\Users\jenkins\.java\java8\bin\java -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\distributions\logger-usage-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.ow2.asm\asm-debug-all\5.0.4\702b8525fcf81454235e5e2fa2a35f15ffc0ec7e\asm-debug-all-5.0.4.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar org.elasticsearch.test.loggerusage.ESLoggerUsageChecker C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\classes\java\main C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\classes\java\test Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' checking for wrong usages of ESLogger... No wrong usages found :libs:secure-sm:loggerUsageCheck (Thread[Daemon worker,5,main]) completed. Took 0.246 secs. :libs:secure-sm:namingConventions (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:secure-sm:namingConventions Task ':libs:secure-sm:namingConventions' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm Command: C:\Users\jenkins\.java\java8\bin\java -Djna.nosys=true -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\distributions\build-tools-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\classes\java\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\resources\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\resources\test org.elasticsearch.test.NamingConventionsCheck --test-class junit.framework.TestCase --integ-test-class org.elasticsearch.test.ESIntegTestCase -- C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\classes\java\test Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' :libs:secure-sm:namingConventions (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.181 secs. :libs:secure-sm:thirdPartyAudit (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:secure-sm:thirdPartyAudit SKIPPED Skipping task ':libs:secure-sm:thirdPartyAudit' as task onlyIf is false. :libs:secure-sm:thirdPartyAudit (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :libs:secure-sm:precommit (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:secure-sm:precommit Skipping task ':libs:secure-sm:precommit' as it has no actions. :libs:secure-sm:precommit (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :libs:secure-sm:test (Thread[Task worker for ':' Thread 3,5,main]) started. Started J0 PID(9076@SLAVE-0633DD87F). Started J1 PID(10304@SLAVE-0633DD87F). Suite: org.elasticsearch.secure_sm.ThreadPermissionTests Completed [1/2] on J1 in 0.01s, 2 tests Suite: org.elasticsearch.secure_sm.SecureSMTests Completed [2/2] on J0 in 0.03s, 6 tests > Task :libs:secure-sm:test Task ':libs:secure-sm:test' is not up-to-date because: Task.upToDateWhen is false. [ant:junit4] says hello! Master seed: 14D96C1B272DD540 ==> Test Info: seed=14D96C1B272DD540; jvms=2; suites=2 Slow Tests Summary: 0.03s | org.elasticsearch.secure_sm.SecureSMTests 0.01s | org.elasticsearch.secure_sm.ThreadPermissionTests ==> Test Summary: 2 suites, 8 tests [ant:junit4] JVM J0: 0.53 .. 0.80 = 0.27s [ant:junit4] JVM J1: 0.53 .. 0.79 = 0.26s [ant:junit4] Execution time total: 0.80 sec. [ant:junit4] Tests summary: 2 suites, 8 tests :libs:secure-sm:test (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.822 secs. :libs:x-content:copyCheckstyleConf (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:copyCheckstyleConf Task ':libs:x-content:copyCheckstyleConf' is not up-to-date because: No history is available. :libs:x-content:copyCheckstyleConf (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.003 secs. :libs:x-content:checkstyleMain (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:checkstyleMain Task ':libs:x-content:checkstyleMain' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 35 files :libs:x-content:checkstyleMain (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.233 secs. :libs:x-content:compileTestJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:compileTestJava Task ':libs:x-content:compileTestJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':libs:x-content:compileTestJava'. :libs:x-content:compileTestJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\tmp\compileTestJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.017 secs. Created jar classpath snapshot for incremental compilation in 0.004 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :libs:x-content:compileTestJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.465 secs. :libs:x-content:processTestResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:processTestResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\src\test\resources', not found Skipping task ':libs:x-content:processTestResources' as it has no source files and no previous output files. :libs:x-content:processTestResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :libs:x-content:testClasses (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:testClasses Skipping task ':libs:x-content:testClasses' as it has no actions. :libs:x-content:testClasses (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :libs:x-content:checkstyleTest (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:checkstyleTest Task ':libs:x-content:checkstyleTest' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 4 files :libs:x-content:checkstyleTest (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.149 secs. :libs:x-content:checkstyle (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:checkstyle Skipping task ':libs:x-content:checkstyle' as it has no actions. :libs:x-content:checkstyle (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :libs:x-content:dependencyLicenses (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:dependencyLicenses Task ':libs:x-content:dependencyLicenses' is not up-to-date because: Task has not declared any outputs despite executing actions. Checking sha for snakeyaml-1.17.jar Checking license/notice for snakeyaml Checking sha for jackson-core-2.8.10.jar Checking license/notice for jackson-core Mapped dependency name jackson-core to jackson for license check Checking sha for jackson-dataformat-smile-2.8.10.jar Checking license/notice for jackson-dataformat-smile Mapped dependency name jackson-dataformat-smile to jackson for license check Checking sha for jackson-dataformat-yaml-2.8.10.jar Checking license/notice for jackson-dataformat-yaml Mapped dependency name jackson-dataformat-yaml to jackson for license check Checking sha for jackson-dataformat-cbor-2.8.10.jar Checking license/notice for jackson-dataformat-cbor Mapped dependency name jackson-dataformat-cbor to jackson for license check :libs:x-content:dependencyLicenses (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.007 secs. :libs:x-content:filepermissions (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:filepermissions SKIPPED Skipping task ':libs:x-content:filepermissions' as task onlyIf is false. :libs:x-content:filepermissions (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :libs:x-content:forbiddenApisMain (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:forbiddenApisMain Task ':libs:x-content:forbiddenApisMain' is not up-to-date because: No history is available. Reading bundled API signatures: jdk-unsafe-1.8 Reading bundled API signatures: jdk-deprecated-1.8 Reading bundled API signatures: jdk-non-portable Reading bundled API signatures: jdk-system-out Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/jdk-signatures.txt Loading classes to check... Scanning classes for violations... Scanned 77 class file(s) for forbidden API invocations (in 0.09s), 0 error(s). :libs:x-content:forbiddenApisMain (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.095 secs. :libs:x-content:forbiddenApisTest (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:forbiddenApisTest Task ':libs:x-content:forbiddenApisTest' is not up-to-date because: No history is available. Reading bundled API signatures: jdk-unsafe-1.8 Reading bundled API signatures: jdk-deprecated-1.8 Reading bundled API signatures: jdk-non-portable Reading bundled API signatures: jdk-system-out Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/jdk-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-all-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-test-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/http-signatures.txt Loading classes to check... Scanning classes for violations... Scanned 27 class file(s) for forbidden API invocations (in 0.07s), 0 error(s). :libs:x-content:forbiddenApisTest (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.074 secs. :libs:x-content:forbiddenApis (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:forbiddenApis Skipping task ':libs:x-content:forbiddenApis' as it has no actions. :libs:x-content:forbiddenApis (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :libs:x-content:forbiddenPatterns (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:forbiddenPatterns Task ':libs:x-content:forbiddenPatterns' is not up-to-date because: No history is available. file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\src\main\resources', not found file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\src\test\resources', not found :libs:x-content:forbiddenPatterns (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.028 secs. :libs:x-content:jarHell (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:jarHell SKIPPED Skipping task ':libs:x-content:jarHell' as task onlyIf is false. :libs:x-content:jarHell (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :libs:x-content:licenseHeaders (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:licenseHeaders Task ':libs:x-content:licenseHeaders' is not up-to-date because: No history is available. :libs:x-content:licenseHeaders (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.048 secs. :libs:x-content:loggerUsageCheck (Thread[Daemon worker,5,main]) started. > Task :libs:x-content:loggerUsageCheck Task ':libs:x-content:loggerUsageCheck' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content Command: C:\Users\jenkins\.java\java8\bin\java -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\distributions\logger-usage-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.ow2.asm\asm-debug-all\5.0.4\702b8525fcf81454235e5e2fa2a35f15ffc0ec7e\asm-debug-all-5.0.4.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar org.elasticsearch.test.loggerusage.ESLoggerUsageChecker C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\classes\java\main C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\classes\java\test Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' checking for wrong usages of ESLogger... No wrong usages found :libs:x-content:loggerUsageCheck (Thread[Daemon worker,5,main]) completed. Took 0.372 secs. :libs:x-content:namingConventions (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:namingConventions Task ':libs:x-content:namingConventions' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content Command: C:\Users\jenkins\.java\java8\bin\java -Djna.nosys=true -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\distributions\build-tools-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\classes\java\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\resources\main;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\resources\test org.elasticsearch.test.NamingConventionsCheck --test-class org.apache.lucene.util.LuceneTestCase --integ-test-class org.elasticsearch.test.ESIntegTestCase -- C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\classes\java\test Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' :libs:x-content:namingConventions (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.198 secs. :libs:x-content:thirdPartyAudit (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:thirdPartyAudit Task ':libs:x-content:thirdPartyAudit' is not up-to-date because: No history is available. [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\tmp\thirdPartyAudit [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\tmp\thirdPartyAudit [thirdPartyAudit] Reading inline API signatures... [thirdPartyAudit] Loading classes to check... [thirdPartyAudit] Scanning classes for violations... [thirdPartyAudit] Scanned 340 class file(s) for forbidden API invocations (in 0.48s), 0 error(s). [delete] Deleting directory C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\tmp\thirdPartyAudit :libs:x-content:thirdPartyAudit (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.806 secs. :libs:x-content:precommit (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:precommit Skipping task ':libs:x-content:precommit' as it has no actions. :libs:x-content:precommit (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :libs:x-content:test (Thread[Task worker for ':' Thread 3,5,main]) started. Started J0 PID(12116@SLAVE-0633DD87F). Started J1 PID(1700@SLAVE-0633DD87F). Started J2 PID(12224@SLAVE-0633DD87F). > Task :libs:x-content:test Task ':libs:x-content:test' is not up-to-date because: Task.upToDateWhen is false. [ant:junit4] says ???! Master seed: 94522C193B8F7A8F ==> Test Info: seed=94522C193B8F7A8F; jvms=3; suites=4 Suite: org.elasticsearch.common.ParseFieldTests Completed [1/4] on J0 in 4.06s, 3 tests Suite: org.elasticsearch.common.xcontent.ConstructingObjectParserTests IGNOR/A 0.02s J1 | ConstructingObjectParserTests.testRepeatedConstructorParam > Assumption #1: Test only makes sense if XContent parser doesn't have strict duplicate checks enabled Completed [2/4] on J1 in 4.23s, 25 tests, 1 skipped Suite: org.elasticsearch.common.xcontent.ObjectParserTests Completed [3/4] on J2 in 4.52s, 23 tests Suite: org.elasticsearch.common.xcontent.XContentParserTests Completed [4/4] on J0 in 0.51s, 11 tests > Task :libs:x-content:test Slow Tests Summary: 4.52s | org.elasticsearch.common.xcontent.ObjectParserTests 4.23s | org.elasticsearch.common.xcontent.ConstructingObjectParserTests 4.06s | org.elasticsearch.common.ParseFieldTests 0.51s | org.elasticsearch.common.xcontent.XContentParserTests ==> Test Summary: 4 suites, 62 tests, 1 ignored (1 assumption) [ant:junit4] JVM J0: 0.54 .. 5.95 = 5.42s [ant:junit4] JVM J1: 0.54 .. 5.88 = 5.34s [ant:junit4] JVM J2: 0.54 .. 5.89 = 5.34s [ant:junit4] Execution time total: 5.96 sec. [ant:junit4] Tests summary: 4 suites, 62 tests, 1 ignored (1 assumption) :libs:x-content:test (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 5.975 secs. :server:cli:copyCheckstyleConf (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:copyCheckstyleConf Task ':server:cli:copyCheckstyleConf' is not up-to-date because: No history is available. :server:cli:copyCheckstyleConf (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.004 secs. :server:cli:checkstyleMain (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:checkstyleMain Task ':server:cli:checkstyleMain' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 6 files :server:cli:checkstyleMain (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.083 secs. :server:cli:compileTestJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:compileTestJava NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\src\test\java', not found Skipping task ':server:cli:compileTestJava' as it has no source files and no previous output files. :server:cli:compileTestJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :server:cli:processTestResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:processTestResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\src\test\resources', not found Skipping task ':server:cli:processTestResources' as it has no source files and no previous output files. :server:cli:processTestResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :server:cli:testClasses (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:testClasses UP-TO-DATE Skipping task ':server:cli:testClasses' as it has no actions. :server:cli:testClasses (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :server:cli:checkstyleTest (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:checkstyleTest NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\src\test\java', not found Skipping task ':server:cli:checkstyleTest' as it has no source files and no previous output files. :server:cli:checkstyleTest (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :server:cli:checkstyle (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:checkstyle Skipping task ':server:cli:checkstyle' as it has no actions. :server:cli:checkstyle (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :server:cli:dependencyLicenses (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:dependencyLicenses Task ':server:cli:dependencyLicenses' is not up-to-date because: Task has not declared any outputs despite executing actions. Checking sha for jopt-simple-5.0.2.jar Checking license/notice for jopt-simple :server:cli:dependencyLicenses (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.002 secs. :server:cli:filepermissions (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:filepermissions SKIPPED Skipping task ':server:cli:filepermissions' as task onlyIf is false. :server:cli:filepermissions (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :server:cli:forbiddenApisMain (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:forbiddenApisMain Task ':server:cli:forbiddenApisMain' is not up-to-date because: No history is available. Reading bundled API signatures: jdk-unsafe-1.8 Reading bundled API signatures: jdk-deprecated-1.8 Reading bundled API signatures: jdk-non-portable Reading bundled API signatures: jdk-system-out Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/jdk-signatures.txt Loading classes to check... Scanning classes for violations... Scanned 9 class file(s) for forbidden API invocations (in 0.04s), 0 error(s). :server:cli:forbiddenApisMain (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.046 secs. :server:cli:forbiddenApisTest (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:forbiddenApisTest NO-SOURCE Skipping task ':server:cli:forbiddenApisTest' as it has no source files and no previous output files. :server:cli:forbiddenApisTest (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :server:cli:forbiddenApis (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:forbiddenApis Skipping task ':server:cli:forbiddenApis' as it has no actions. :server:cli:forbiddenApis (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :server:cli:forbiddenPatterns (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:forbiddenPatterns Task ':server:cli:forbiddenPatterns' is not up-to-date because: No history is available. file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\src\main\resources', not found file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\src\test\resources', not found file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\src\test\java', not found :server:cli:forbiddenPatterns (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.006 secs. :server:cli:jarHell (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:jarHell SKIPPED Skipping task ':server:cli:jarHell' as task onlyIf is false. :server:cli:jarHell (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :server:cli:licenseHeaders (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:licenseHeaders file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\src\test\java', not found Task ':server:cli:licenseHeaders' is not up-to-date because: No history is available. :server:cli:licenseHeaders (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.042 secs. :server:cli:loggerUsageCheck (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:loggerUsageCheck Task ':server:cli:loggerUsageCheck' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli Command: C:\Users\jenkins\.java\java8\bin\java -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\distributions\logger-usage-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.ow2.asm\asm-debug-all\5.0.4\702b8525fcf81454235e5e2fa2a35f15ffc0ec7e\asm-debug-all-5.0.4.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar org.elasticsearch.test.loggerusage.ESLoggerUsageChecker C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\classes\java\main Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' checking for wrong usages of ESLogger... No wrong usages found :server:cli:loggerUsageCheck (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.25 secs. :server:cli:namingConventions (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:namingConventions SKIPPED Skipping task ':server:cli:namingConventions' as task onlyIf is false. :server:cli:namingConventions (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :server:cli:thirdPartyAudit (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:thirdPartyAudit Task ':server:cli:thirdPartyAudit' is not up-to-date because: No history is available. [unzip] Expanding: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar into C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\tmp\thirdPartyAudit [thirdPartyAudit] Reading inline API signatures... [thirdPartyAudit] Loading classes to check... [thirdPartyAudit] Scanning classes for violations... [thirdPartyAudit] Scanned 59 class file(s) for forbidden API invocations (in 0.07s), 0 error(s). [delete] Deleting directory C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\tmp\thirdPartyAudit :server:cli:thirdPartyAudit (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.137 secs. :server:cli:precommit (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:precommit Skipping task ':server:cli:precommit' as it has no actions. :server:cli:precommit (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :server:cli:test (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:test SKIPPED Skipping task ':server:cli:test' as task onlyIf is false. :server:cli:test (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :server:test (Thread[Task worker for ':' Thread 3,5,main]) started. Started J2 PID(11172@SLAVE-0633DD87F). Started J0 PID(11392@SLAVE-0633DD87F). Started J1 PID(6380@SLAVE-0633DD87F). > Task :server:test Task ':server:test' is not up-to-date because: Task.upToDateWhen is false. [ant:junit4] says ol?! Master seed: ED566F0D57EEE89A ==> Test Info: seed=ED566F0D57EEE89A; jvms=3; suites=1066 Suite: org.elasticsearch.action.admin.cluster.snapshots.status.SnapshotStatusTests Completed [1/1066] on J2 in 5.61s, 1 test Suite: org.elasticsearch.search.suggest.SuggestionEntryTests Completed [2/1066] on J2 in 1.57s, 3 tests Suite: org.elasticsearch.action.admin.indices.alias.AliasActionsTests Completed [3/1066] on J2 in 0.50s, 12 tests Suite: org.elasticsearch.search.suggest.SuggestBuilderTests Completed [4/1066] on J0 in 8.16s, 4 tests Suite: org.elasticsearch.action.admin.cluster.shards.ClusterSearchShardsResponseTests Completed [5/1066] on J1 in 8.21s, 1 test Suite: org.elasticsearch.cluster.routing.GroupShardsIteratorTests Completed [6/1066] on J0 in 0.35s, 2 tests Suite: org.elasticsearch.search.suggest.SuggestTests Completed [7/1066] on J1 in 1.32s, 6 tests Suite: org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregationBuilderTests Completed [8/1066] on J2 in 3.33s, 5 tests Suite: org.elasticsearch.action.search.MultiSearchActionTookTests Completed [9/1066] on J0 in 2.76s, 2 tests Suite: org.elasticsearch.action.ExplainRequestTests Completed [10/1066] on J0 in 0.31s, 1 test Suite: org.elasticsearch.search.aggregations.InternalOrderTests Completed [11/1066] on J0 in 0.21s, 5 tests Suite: org.elasticsearch.gateway.MetaDataStateFormatTests Completed [12/1066] on J2 in 1.17s, 5 tests Suite: org.elasticsearch.search.aggregations.metrics.ScriptedMetricTests Completed [13/1066] on J2 in 0.12s, 5 tests Suite: org.elasticsearch.search.SearchModuleTests Completed [14/1066] on J2 in 0.08s, 7 tests Suite: org.elasticsearch.discovery.zen.MembershipActionTests Completed [15/1066] on J2 in 0.16s, 4 tests Suite: org.elasticsearch.search.fetch.subphase.highlight.HighlightFieldTests Completed [16/1066] on J2 in 0.11s, 4 tests Suite: org.elasticsearch.cluster.service.TaskExecutorTests Completed [17/1066] on J2 in 0.45s, 3 tests Suite: org.elasticsearch.index.MergeSchedulerSettingsTests Completed [18/1066] on J2 in 0.30s, 3 tests Suite: org.elasticsearch.action.admin.indices.TransportAnalyzeActionTests Completed [19/1066] on J1 in 3.95s, 9 tests Suite: org.elasticsearch.search.aggregations.bucket.adjacency.InternalAdjacencyMatrixTests Completed [20/1066] on J1 in 0.44s, 6 tests Suite: org.elasticsearch.gateway.GatewayServiceTests Completed [21/1066] on J1 in 0.75s, 1 test Suite: org.elasticsearch.search.aggregations.metrics.PercentilesTests Completed [22/1066] on J1 in 0.65s, 7 tests Suite: org.elasticsearch.search.SearchHitsTests Completed [23/1066] on J1 in 0.18s, 3 tests Suite: org.elasticsearch.discovery.zen.ElectMasterServiceTests Completed [24/1066] on J1 in 0.13s, 6 tests Suite: org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilderTests Completed [25/1066] on J1 in 0.39s, 15 tests Suite: org.elasticsearch.cluster.service.TaskBatcherTests Completed [26/1066] on J1 in 0.95s, 7 tests Suite: org.elasticsearch.index.MergePolicySettingsTests Completed [27/1066] on J1 in 0.12s, 4 tests Suite: org.elasticsearch.index.shard.SearchOperationListenerTests Completed [28/1066] on J1 in 0.19s, 1 test Suite: org.elasticsearch.gateway.AsyncShardFetchTests Completed [29/1066] on J1 in 0.22s, 8 tests Suite: org.elasticsearch.index.shard.ShardGetServiceTests Completed [30/1066] on J2 in 6.21s, 2 tests Suite: org.elasticsearch.indices.analysis.AnalysisModuleTests Completed [31/1066] on J1 in 1.91s, 10 tests Suite: org.elasticsearch.index.reindex.BulkByScrollTaskStatusTests Completed [32/1066] on J1 in 0.15s, 1 test Suite: org.elasticsearch.common.network.NetworkModuleTests Completed [33/1066] on J1 in 0.20s, 5 tests Suite: org.elasticsearch.search.profile.TimerTests Completed [34/1066] on J1 in 0.09s, 2 tests Suite: org.elasticsearch.common.xcontent.support.filtering.JsonXContentFilteringTests Completed [35/1066] on J1 in 0.45s, 36 tests Suite: org.elasticsearch.index.query.SpanTermQueryBuilderTests Completed [36/1066] on J1 in 0.79s, 13 tests Suite: org.elasticsearch.gateway.DanglingIndicesStateTests Completed [37/1066] on J2 in 1.98s, 5 tests HEARTBEAT J0 PID(11392@SLAVE-0633DD87F): 2018-05-05T08:27:39, stalled for 11.3s at: TermVectorsServiceTests.testTook Suite: org.elasticsearch.rest.RestControllerTests Completed [38/1066] on J1 in 1.96s, 20 tests Suite: org.elasticsearch.indices.analyze.HunspellServiceTests Completed [39/1066] on J2 in 2.77s, 4 tests Suite: org.elasticsearch.index.reindex.BulkByScrollTaskTests Completed [40/1066] on J2 in 0.29s, 5 tests Suite: org.elasticsearch.ingest.IngestServiceTests Completed [41/1066] on J1 in 1.19s, 2 tests Suite: org.elasticsearch.index.termvectors.TermVectorsServiceTests Completed [42/1066] on J0 in 13.41s, 2 tests Suite: org.elasticsearch.rest.action.document.RestIndexActionTests Completed [43/1066] on J0 in 0.05s, 1 test Suite: org.elasticsearch.common.network.NetworkServiceTests Completed [44/1066] on J2 in 0.66s, 10 tests Suite: org.elasticsearch.search.profile.aggregation.AggregationProfileShardResultTests Completed [45/1066] on J2 in 0.35s, 2 tests Suite: org.elasticsearch.deps.lucene.SimpleLuceneTests Completed [46/1066] on J1 in 1.51s, 4 tests Suite: org.elasticsearch.common.xcontent.support.filtering.SmileFilteringGeneratorTests Completed [47/1066] on J2 in 0.61s, 36 tests Suite: org.elasticsearch.cluster.metadata.AutoExpandReplicasTests Completed [48/1066] on J1 in 0.20s, 2 tests Suite: org.elasticsearch.index.query.SpanWithinQueryBuilderTests Completed [49/1066] on J2 in 0.73s, 11 tests Suite: org.elasticsearch.rest.RestHttpResponseHeadersTests Completed [50/1066] on J2 in 0.17s, 1 test Suite: org.elasticsearch.ingest.IngestStatsTests Completed [51/1066] on J2 in 0.09s, 1 test Suite: org.elasticsearch.index.IndexModuleTests Completed [52/1066] on J1 in 1.34s, 13 tests Suite: org.elasticsearch.search.aggregations.metrics.StatsTests Completed [53/1066] on J1 in 0.09s, 5 tests Suite: org.elasticsearch.action.fieldcaps.FieldCapabilitiesTests Completed [54/1066] on J1 in 0.11s, 4 tests Suite: org.elasticsearch.deps.lucene.VectorHighlighterTests Completed [55/1066] on J2 in 0.62s, 4 tests Suite: org.elasticsearch.index.search.nested.LongNestedSortingTests Completed [56/1066] on J0 in 2.82s, 2 tests Suite: org.elasticsearch.action.search.AbstractSearchAsyncActionTests Completed [57/1066] on J1 in 0.22s, 3 tests Suite: org.elasticsearch.cluster.metadata.ClusterNameExpressionResolverTests Completed [58/1066] on J2 in 0.25s, 7 tests Suite: org.elasticsearch.action.search.SearchScrollRequestTests Completed [59/1066] on J0 in 0.20s, 7 tests Suite: org.elasticsearch.common.compress.DeflateCompressTests Completed [60/1066] on J1 in 2.51s, 11 tests Suite: org.elasticsearch.action.admin.indices.settings.get.GetSettingsActionTests Completed [61/1066] on J1 in 0.41s, 2 tests Suite: org.elasticsearch.search.NestedIdentityTests Completed [62/1066] on J1 in 0.05s, 4 tests Suite: org.elasticsearch.cluster.service.ClusterApplierServiceTests Completed [63/1066] on J1 in 0.16s, 5 tests Suite: org.elasticsearch.indices.recovery.RecoveryStatusTests Completed [64/1066] on J1 in 4.59s, 1 test Suite: org.elasticsearch.search.aggregations.bucket.range.InternalGeoDistanceTests Completed [65/1066] on J1 in 0.22s, 6 tests Suite: org.elasticsearch.index.IndexServiceTests Completed [66/1066] on J2 in 8.62s, 8 tests Suite: org.elasticsearch.search.aggregations.metrics.SumAggregatorTests Completed [67/1066] on J2 in 0.81s, 7 tests Suite: org.elasticsearch.action.get.GetResponseTests Completed [68/1066] on J2 in 0.71s, 6 tests Suite: org.elasticsearch.action.search.CanMatchPreFilterSearchPhaseTests Completed [69/1066] on J2 in 0.25s, 4 tests HEARTBEAT J0 PID(11392@SLAVE-0633DD87F): 2018-05-05T08:27:56, stalled for 10.7s at: RefreshListenersTests.testConcurrentRefresh Suite: org.elasticsearch.common.compress.DeflateCompressedXContentTests Completed [70/1066] on J2 in 1.41s, 3 tests Suite: org.elasticsearch.action.admin.indices.settings.get.GetSettingsRequestTests Completed [71/1066] on J2 in 0.13s, 3 tests Suite: org.elasticsearch.search.SearchCancellationTests Completed [72/1066] on J2 in 0.05s, 1 test Suite: org.elasticsearch.cluster.service.ClusterSerivceTests Completed [73/1066] on J2 in 0.05s, 1 test Suite: org.elasticsearch.indices.recovery.RecoveryTargetTests Completed [74/1066] on J2 in 0.15s, 7 tests Suite: org.elasticsearch.index.similarity.SimilarityTests Completed [75/1066] on J1 in 4.73s, 13 tests Suite: org.elasticsearch.search.aggregations.bucket.range.InternalRangeTests Completed [76/1066] on J2 in 1.20s, 6 tests Suite: org.elasticsearch.index.query.GeoPolygonQueryBuilderTests IGNOR/A 0.05s J1 | GeoPolygonQueryBuilderTests.testMustRewrite > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.00s J1 | GeoPolygonQueryBuilderTests.testParsingAndToQuery3 > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.00s J1 | GeoPolygonQueryBuilderTests.testParsingAndToQuery1 > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.00s J1 | GeoPolygonQueryBuilderTests.testToQuery > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.12s J1 | GeoPolygonQueryBuilderTests.testParsingAndToQuery4 > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.02s J1 | GeoPolygonQueryBuilderTests.testParsingAndToQuery2 > Assumption #1: test runs only when at least a type is registered Completed [77/1066] on J1 in 1.22s, 20 tests, 6 skipped Suite: org.elasticsearch.index.snapshots.blobstore.FileInfoTests Completed [78/1066] on J2 in 0.69s, 3 tests Suite: org.elasticsearch.index.query.GeoShapeQueryBuilderTests IGNOR/A 0.01s J2 | GeoShapeQueryBuilderTests.testWrongFieldType > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.00s J2 | GeoShapeQueryBuilderTests.testToQuery > Assumption #1: test runs only when at least a type is registered Completed [79/1066] on J2 in 0.84s, 21 tests, 2 skipped Suite: org.elasticsearch.index.shard.RefreshListenersTests Completed [80/1066] on J0 in 17.10s, 7 tests Suite: org.elasticsearch.transport.TransportServiceHandshakeTests Completed [81/1066] on J0 in 0.45s, 4 tests Suite: org.elasticsearch.index.similarity.SimilarityServiceTests Completed [82/1066] on J0 in 0.03s, 3 tests Suite: org.elasticsearch.index.analysis.PathHierarchyTokenizerFactoryTests Completed [83/1066] on J0 in 0.03s, 6 tests Suite: org.elasticsearch.common.unit.FuzzinessTests Completed [84/1066] on J0 in 0.04s, 7 tests Suite: org.elasticsearch.common.geo.GeoDistanceTests Completed [85/1066] on J0 in 0.10s, 5 tests Suite: org.elasticsearch.discovery.zen.ZenDiscoveryUnitTests Completed [86/1066] on J2 in 4.20s, 6 tests Suite: org.elasticsearch.index.seqno.LocalCheckpointTrackerTests Completed [87/1066] on J2 in 0.09s, 8 tests Suite: org.apache.lucene.queries.LongRandomBinaryDocValuesRangeQueryTests IGNOR/A 0.02s J0 | LongRandomBinaryDocValuesRangeQueryTests.testRandomBig > Assumption #1: 'nightly' test group is disabled (@Nightly()) Completed [88/1066] on J0 in 1.59s, 4 tests, 1 skipped Suite: org.elasticsearch.search.aggregations.pipeline.bucketmetrics.MaxBucketTests Completed [89/1066] on J2 in 0.11s, 4 tests Suite: org.elasticsearch.search.aggregations.bucket.range.IpRangeAggregatorTests Completed [90/1066] on J0 in 0.45s, 1 test Suite: org.elasticsearch.common.ChannelsTests Completed [91/1066] on J2 in 0.22s, 6 tests Suite: org.elasticsearch.cluster.routing.RecoverySourceTests Completed [92/1066] on J2 in 0.03s, 2 tests Suite: org.elasticsearch.action.NotifyOnceListenerTests Completed [93/1066] on J2 in 0.05s, 2 tests Suite: org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutorTests Completed [94/1066] on J0 in 0.17s, 6 tests Suite: org.elasticsearch.index.query.MultiMatchQueryBuilderTests Completed [95/1066] on J2 in 0.48s, 23 tests Suite: org.elasticsearch.common.network.InetAddressesTests Completed [96/1066] on J2 in 0.05s, 11 tests Suite: org.elasticsearch.indices.recovery.RecoverySettingsDynamicUpdateTests Completed [97/1066] on J2 in 0.08s, 6 tests Suite: org.elasticsearch.common.ReleasablesTests Completed [98/1066] on J2 in 0.04s, 1 test Suite: org.elasticsearch.search.aggregations.metrics.ExtendedStatsTests Completed [99/1066] on J2 in 0.07s, 5 tests Suite: org.elasticsearch.action.DocWriteResponseTests Completed [100/1066] on J2 in 0.05s, 4 tests Suite: org.elasticsearch.indices.memory.breaker.CircuitBreakerUnitTests Completed [101/1066] on J2 in 0.04s, 3 tests Suite: org.elasticsearch.search.aggregations.metrics.InternalStatsBucketTests Completed [102/1066] on J2 in 0.07s, 7 tests Suite: org.elasticsearch.action.search.SearchPhaseExecutionExceptionTests Completed [103/1066] on J2 in 0.05s, 2 tests Suite: org.elasticsearch.index.SearchSlowLogTests Completed [104/1066] on J0 in 1.10s, 4 tests Suite: org.elasticsearch.cluster.ClusterModuleTests Completed [105/1066] on J0 in 0.30s, 12 tests Suite: org.elasticsearch.discovery.zen.UnicastZenPingTests Completed [106/1066] on J1 in 7.63s, 9 tests Suite: org.elasticsearch.script.ScriptTests Completed [107/1066] on J0 in 0.09s, 3 tests Suite: org.elasticsearch.index.seqno.GlobalCheckpointSyncActionTests Completed [108/1066] on J1 in 0.14s, 1 test Suite: org.elasticsearch.search.aggregations.pipeline.bucketmetrics.InternalBucketMetricValueTests Completed [109/1066] on J1 in 0.09s, 5 tests Suite: org.elasticsearch.common.BooleansTests Completed [110/1066] on J1 in 0.15s, 8 tests Suite: org.elasticsearch.cluster.routing.PrimaryTermsTests Completed [111/1066] on J1 in 0.13s, 1 test Suite: org.elasticsearch.action.GenericActionTests Completed [112/1066] on J1 in 0.07s, 1 test Suite: org.elasticsearch.index.query.MoreLikeThisQueryBuilderTests IGNOR/A 0.00s J1 | MoreLikeThisQueryBuilderTests.testUnsupportedFields > Assumption #1: test runs only when at least a type is registered Completed [113/1066] on J1 in 0.36s, 18 tests, 1 skipped Suite: org.elasticsearch.index.mapper.KeywordFieldMapperTests Completed [114/1066] on J2 in 2.46s, 12 tests Suite: org.elasticsearch.action.admin.indices.rollover.ConditionTests Completed [115/1066] on J2 in 0.04s, 4 tests Suite: org.elasticsearch.search.suggest.completion.CompletionSuggesterBuilderTests Completed [116/1066] on J2 in 0.37s, 4 tests Suite: org.elasticsearch.action.admin.cluster.storedscripts.GetStoredScriptRequestTests Completed [117/1066] on J2 in 0.06s, 1 test Suite: org.elasticsearch.action.admin.indices.stats.IndicesStatsResponseTests Completed [118/1066] on J2 in 0.03s, 1 test Suite: org.elasticsearch.action.search.MultiSearchResponseTests Completed [119/1066] on J2 in 0.25s, 1 test Suite: org.elasticsearch.common.network.CidrsTests Completed [120/1066] on J1 in 2.05s, 9 tests Suite: org.elasticsearch.indices.recovery.PeerRecoveryTargetServiceTests Completed [121/1066] on J1 in 0.54s, 1 test Suite: org.elasticsearch.common.PriorityTests Completed [122/1066] on J1 in 0.03s, 4 tests Suite: org.elasticsearch.action.admin.cluster.node.tasks.CancellableTasksTests Completed [123/1066] on J0 in 3.86s, 3 tests Suite: org.elasticsearch.index.analysis.CustomNormalizerTests Completed [124/1066] on J0 in 0.11s, 6 tests Suite: org.elasticsearch.search.aggregations.metrics.ExtendedStatsAggregatorTests Completed [125/1066] on J1 in 0.59s, 4 tests Suite: org.elasticsearch.action.ActionModuleTests Completed [126/1066] on J1 in 0.09s, 6 tests Suite: org.elasticsearch.index.search.NestedHelperTests Completed [127/1066] on J0 in 1.22s, 7 tests Suite: org.elasticsearch.indices.recovery.RecoverySourceHandlerTests Completed [128/1066] on J0 in 0.66s, 6 tests Suite: org.elasticsearch.indices.TermsLookupTests Completed [129/1066] on J0 in 0.05s, 3 tests Suite: org.elasticsearch.action.admin.cluster.settings.ClusterUpdateSettingsRequestTests Completed [130/1066] on J0 in 0.07s, 2 tests Suite: org.elasticsearch.cluster.routing.allocation.decider.DecisionTests Completed [131/1066] on J0 in 0.13s, 1 test Suite: org.elasticsearch.discovery.ZenFaultDetectionTests Completed [132/1066] on J2 in 3.69s, 3 tests Suite: org.elasticsearch.action.bulk.BackoffPolicyTests Completed [133/1066] on J0 in 0.11s, 1 test Suite: org.elasticsearch.search.suggest.SuggestionOptionTests Completed [134/1066] on J0 in 0.08s, 3 tests Suite: org.elasticsearch.action.admin.cluster.health.ClusterHealthRequestTests Completed [135/1066] on J2 in 0.03s, 1 test Suite: org.elasticsearch.cluster.routing.allocation.AwarenessAllocationTests Completed [136/1066] on J0 in 0.71s, 12 tests Suite: org.elasticsearch.index.reindex.UpdateByQueryRequestTests Completed [137/1066] on J0 in 0.08s, 2 tests Suite: org.elasticsearch.index.query.ScoreModeTests Completed [138/1066] on J0 in 0.09s, 4 tests Suite: org.elasticsearch.indices.flush.SyncedFlushSingleNodeTests Completed [139/1066] on J1 in 3.30s, 6 tests Suite: org.elasticsearch.common.util.LongHashTests Completed [140/1066] on J2 in 1.53s, 5 tests Suite: org.elasticsearch.search.aggregations.metrics.InternalMinTests Completed [141/1066] on J1 in 0.13s, 5 tests Suite: org.elasticsearch.common.settings.ScopedSettingsTests Completed [142/1066] on J2 in 0.26s, 27 tests Suite: org.elasticsearch.action.search.SearchPhaseControllerTests Completed [143/1066] on J1 in 0.19s, 9 tests Suite: org.elasticsearch.search.aggregations.AggregationCollectorTests Completed [144/1066] on J0 in 0.89s, 1 test Suite: org.elasticsearch.cluster.routing.allocation.UpdateNumberOfReplicasTests Completed [145/1066] on J2 in 0.05s, 1 test Suite: org.elasticsearch.index.mapper.JavaMultiFieldMergeTests Completed [146/1066] on J1 in 1.15s, 2 tests Suite: org.elasticsearch.action.admin.indices.refresh.RefreshResponseTests Completed [147/1066] on J1 in 0.20s, 3 tests Suite: org.elasticsearch.action.termvectors.GetTermVectorsTests Completed [148/1066] on J0 in 1.47s, 1 test Suite: org.elasticsearch.search.suggest.completion.CategoryQueryContextTests Completed [149/1066] on J1 in 0.09s, 3 tests Suite: org.elasticsearch.action.admin.cluster.stats.ClusterStatsNodesTests Completed [150/1066] on J1 in 0.13s, 1 test Suite: org.elasticsearch.search.suggest.term.StringDistanceImplTests Completed [151/1066] on J0 in 0.19s, 4 tests Suite: org.elasticsearch.action.admin.indices.shrink.TransportResizeActionTests Completed [152/1066] on J1 in 0.13s, 4 tests Suite: org.elasticsearch.action.search.MultiSearchRequestTests Completed [153/1066] on J1 in 0.66s, 10 tests Suite: org.elasticsearch.discovery.DiscoveryModuleTests Completed [154/1066] on J1 in 0.29s, 10 tests Suite: org.elasticsearch.action.admin.cluster.allocation.ClusterAllocationExplanationTests Completed [155/1066] on J1 in 0.38s, 3 tests Suite: org.elasticsearch.common.util.LocaleUtilsTests Completed [156/1066] on J1 in 0.45s, 5 tests Suite: org.elasticsearch.search.query.QueryPhaseTests Completed [157/1066] on J0 in 2.55s, 10 tests Suite: org.apache.lucene.grouping.CollapsingTopDocsCollectorTests Completed [158/1066] on J2 in 4.66s, 7 tests Suite: org.elasticsearch.action.search.ClearScrollControllerTests Completed [159/1066] on J0 in 0.34s, 3 tests Suite: org.elasticsearch.common.util.concurrent.PrioritizedRunnableTests Completed [160/1066] on J2 in 0.04s, 2 tests Suite: org.elasticsearch.bootstrap.ElasticsearchUncaughtExceptionHandlerTests Completed [161/1066] on J0 in 0.04s, 3 tests Suite: org.elasticsearch.common.util.ByteUtilsTests Completed [162/1066] on J2 in 0.06s, 4 tests Suite: org.elasticsearch.search.aggregations.metrics.MinTests Completed [163/1066] on J2 in 0.37s, 5 tests Suite: org.elasticsearch.common.settings.SettingsModuleTests Completed [164/1066] on J0 in 0.59s, 6 tests Suite: org.elasticsearch.search.aggregations.metrics.PercentileRanksTests Completed [165/1066] on J0 in 0.10s, 5 tests Suite: org.elasticsearch.search.aggregations.bucket.GlobalTests Completed [166/1066] on J0 in 0.06s, 5 tests Suite: org.elasticsearch.index.query.functionscore.FunctionScoreQueryBuilderTests IGNOR/A 0.02s J2 | FunctionScoreQueryBuilderTests.testWeight1fStillProducesWeightFunction > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.01s J2 | FunctionScoreQueryBuilderTests.testToQuery > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.00s J2 | FunctionScoreQueryBuilderTests.testMustRewrite > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.01s J2 | FunctionScoreQueryBuilderTests.testMalformedQueryMultipleQueryElements > Assumption #1: Test only makes sense if XContent parser doesn't have strict duplicate checks enabled Completed [167/1066] on J2 in 0.40s, 29 tests, 4 skipped Suite: org.elasticsearch.search.aggregations.bucket.MissingTests Completed [168/1066] on J0 in 0.08s, 5 tests Suite: org.elasticsearch.common.settings.RemoveSettingKeyStoreCommandTests Completed [169/1066] on J1 in 2.51s, 5 tests Suite: org.elasticsearch.search.aggregations.AggregatorFactoriesTests IGNOR/A 0.06s J2 | AggregatorFactoriesTests.testSameAggregationName > Assumption #1: Test only makes sense if XContent parser doesn't have strict duplicate checks enabled IGNOR/A 0.01s J2 | AggregatorFactoriesTests.testTwoAggs > Assumption #1: Test only makes sense if XContent parser doesn't have strict duplicate checks enabled Completed [170/1066] on J2 in 0.23s, 9 tests, 2 skipped Suite: org.elasticsearch.index.query.MatchNoneQueryBuilderTests Completed [171/1066] on J0 in 0.30s, 10 tests Suite: org.elasticsearch.action.search.FetchSearchPhaseTests Completed [172/1066] on J2 in 0.08s, 6 tests Suite: org.elasticsearch.client.ParentTaskAssigningClientTests Completed [173/1066] on J2 in 0.03s, 1 test Suite: org.elasticsearch.cluster.routing.allocation.ThrottlingAllocationTests Completed [174/1066] on J1 in 0.23s, 4 tests Suite: org.apache.lucene.analysis.miscellaneous.TruncateTokenFilterTests Completed [175/1066] on J1 in 0.05s, 1 test Suite: org.elasticsearch.search.fetch.subphase.highlight.PlainHighlighterTests Completed [176/1066] on J0 in 0.27s, 1 test Suite: org.elasticsearch.action.admin.cluster.storedscripts.PutStoredScriptRequestTests Completed [177/1066] on J0 in 0.06s, 1 test Suite: org.elasticsearch.common.util.concurrent.PrioritizedExecutorsTests Completed [178/1066] on J1 in 0.30s, 7 tests Suite: org.elasticsearch.common.network.NetworkUtilsTests Completed [179/1066] on J0 in 0.22s, 6 tests Suite: org.elasticsearch.search.aggregations.bucket.SignificantTermsTests Completed [180/1066] on J0 in 0.06s, 5 tests Suite: org.elasticsearch.search.suggest.completion.CompletionSuggestionTests Completed [181/1066] on J0 in 0.23s, 1 test Suite: org.elasticsearch.common.util.ArrayUtilsTests Completed [182/1066] on J0 in 0.17s, 2 tests Suite: org.elasticsearch.nodesinfo.NodeInfoStreamingTests Completed [183/1066] on J0 in 0.13s, 1 test Suite: org.elasticsearch.common.util.LongObjectHashMapTests Completed [184/1066] on J0 in 1.13s, 1 test Suite: org.elasticsearch.discovery.BlockingClusterStatePublishResponseHandlerTests Completed [185/1066] on J0 in 0.06s, 1 test Suite: org.elasticsearch.cluster.routing.allocation.PreferLocalPrimariesToRelocatingPrimariesTests Completed [186/1066] on J0 in 0.13s, 1 test Suite: org.elasticsearch.deps.joda.SimpleJodaTests Completed [187/1066] on J0 in 0.16s, 24 tests Suite: org.elasticsearch.transport.PublishPortTests Completed [188/1066] on J0 in 0.11s, 1 test Suite: org.elasticsearch.action.search.DfsQueryPhaseTests Completed [189/1066] on J0 in 0.09s, 3 tests Suite: org.elasticsearch.action.ingest.SimulatePipelineRequestParsingTests Completed [190/1066] on J0 in 0.09s, 4 tests Suite: org.elasticsearch.search.aggregations.bucket.adjacency.AdjacencyMatrixAggregationBuilderTests Completed [191/1066] on J0 in 0.05s, 1 test Suite: org.elasticsearch.action.admin.indices.mapping.get.GetFieldMappingsResponseTests Completed [192/1066] on J0 in 0.03s, 1 test Suite: org.elasticsearch.cluster.routing.allocation.decider.EnableAllocationTests Completed [193/1066] on J0 in 0.06s, 5 tests Suite: org.elasticsearch.indices.recovery.PeerRecoverySourceServiceTests Completed [194/1066] on J0 in 0.18s, 1 test Suite: org.elasticsearch.index.mapper.NumberFieldMapperTests Completed [195/1066] on J2 in 3.60s, 12 tests Suite: org.elasticsearch.search.suggest.CompletionSuggestionOptionTests Completed [196/1066] on J2 in 0.13s, 3 tests Suite: org.elasticsearch.action.admin.indices.forcemerge.ForceMergeResponseTests Completed [197/1066] on J2 in 0.15s, 3 tests Suite: org.elasticsearch.common.util.BigArraysTests Completed [198/1066] on J1 in 3.20s, 17 tests Suite: org.elasticsearch.search.aggregations.metrics.geobounds.GeoBoundsAggregatorTests Completed [199/1066] on J2 in 0.15s, 2 tests Suite: org.elasticsearch.common.util.URIPatternTests Completed [200/1066] on J2 in 0.08s, 1 test Suite: org.elasticsearch.search.aggregations.metrics.MinAggregatorTests Completed [201/1066] on J1 in 0.34s, 4 tests Suite: org.elasticsearch.index.mapper.DocumentMapperParserTests Completed [202/1066] on J0 in 1.18s, 4 tests Suite: org.elasticsearch.search.suggest.phrase.PhraseSuggestionBuilderTests Completed [203/1066] on J0 in 0.31s, 5 tests Suite: org.elasticsearch.cluster.metadata.IndexCreationTaskTests Completed [204/1066] on J2 in 0.67s, 12 tests Suite: org.elasticsearch.common.io.StreamsTests Completed [205/1066] on J0 in 0.13s, 5 tests Suite: org.elasticsearch.action.bulk.BulkResponseTests Completed [206/1066] on J2 in 0.12s, 1 test Suite: org.elasticsearch.node.InternalSettingsPreparerTests Completed [207/1066] on J0 in 0.31s, 7 tests Suite: org.elasticsearch.action.admin.indices.template.put.PutIndexTemplateRequestTests Completed [208/1066] on J2 in 0.19s, 4 tests Suite: org.elasticsearch.index.query.functionscore.FunctionScoreEquivalenceTests Completed [209/1066] on J1 in 1.02s, 3 tests Suite: org.elasticsearch.action.support.ActiveShardCountTests Completed [210/1066] on J2 in 0.11s, 10 tests Suite: org.elasticsearch.index.IndexTests Completed [211/1066] on J2 in 0.07s, 3 tests Suite: org.elasticsearch.indices.IndicesRequestCacheTests Completed [212/1066] on J0 in 0.61s, 6 tests Suite: org.elasticsearch.search.aggregations.bucket.sampler.DiversifiedAggregationBuilderTests Completed [213/1066] on J2 in 0.60s, 5 tests Suite: org.elasticsearch.search.aggregations.AggregationsTests Completed [214/1066] on J1 in 0.79s, 4 tests Suite: org.elasticsearch.cluster.SnapshotsInProgressTests Completed [215/1066] on J2 in 0.08s, 1 test Suite: org.elasticsearch.index.query.DisMaxQueryBuilderTests Completed [216/1066] on J0 in 0.37s, 12 tests Suite: org.elasticsearch.script.ScriptContextTests Completed [217/1066] on J2 in 0.09s, 6 tests Suite: org.elasticsearch.action.search.ExpandSearchPhaseTests Completed [218/1066] on J1 in 0.15s, 5 tests Suite: org.elasticsearch.cli.TerminalTests Completed [219/1066] on J1 in 0.14s, 5 tests Suite: org.elasticsearch.monitor.jvm.JvmStatsTests Completed [220/1066] on J2 in 0.10s, 1 test Suite: org.elasticsearch.action.support.nodes.TransportNodesActionTests Completed [221/1066] on J2 in 0.15s, 5 tests Suite: org.elasticsearch.cluster.routing.allocation.RebalanceAfterActiveTests Completed [222/1066] on J2 in 0.11s, 1 test Suite: org.elasticsearch.search.aggregations.bucket.histogram.HistogramAggregatorTests Completed [223/1066] on J2 in 0.61s, 7 tests Suite: org.elasticsearch.index.mapper.NullValueTests Completed [224/1066] on J1 in 1.14s, 1 test Suite: org.elasticsearch.action.search.SearchScrollAsyncActionTests Completed [225/1066] on J2 in 0.32s, 5 tests Suite: org.elasticsearch.search.sort.SortOrderTests Completed [226/1066] on J1 in 0.05s, 3 tests Suite: org.elasticsearch.action.admin.indices.flush.SyncedFlushUnitTests Completed [227/1066] on J1 in 0.21s, 2 tests Suite: org.elasticsearch.cli.CommandTests Completed [228/1066] on J2 in 0.34s, 7 tests Suite: org.apache.lucene.queries.FloatRandomBinaryDocValuesRangeQueryTests IGNOR/A 0.00s J0 | FloatRandomBinaryDocValuesRangeQueryTests.testRandomBig > Assumption #1: 'nightly' test group is disabled (@Nightly()) Completed [229/1066] on J0 in 1.82s, 4 tests, 1 skipped Suite: org.elasticsearch.search.aggregations.metrics.cardinality.InternalCardinalityTests Completed [230/1066] on J1 in 0.53s, 5 tests Suite: org.elasticsearch.common.util.SingleObjectCacheTests Completed [231/1066] on J1 in 0.43s, 2 tests Suite: org.elasticsearch.index.shard.ShardSplittingQueryTests Completed [232/1066] on J2 in 0.94s, 4 tests Suite: org.elasticsearch.cluster.metadata.HumanReadableIndexSettingsTests Completed [233/1066] on J1 in 0.11s, 1 test Suite: org.elasticsearch.common.io.stream.NamedWriteableRegistryTests Completed [234/1066] on J2 in 0.03s, 5 tests Suite: org.elasticsearch.common.xcontent.XContentParserUtilsTests Completed [235/1066] on J2 in 0.09s, 14 tests Suite: org.elasticsearch.action.bulk.BulkRequestTests Completed [236/1066] on J1 in 0.13s, 20 tests Suite: org.elasticsearch.indices.IndicesLifecycleListenerSingleNodeTests Completed [237/1066] on J0 in 1.28s, 1 test Suite: org.elasticsearch.index.fielddata.ScriptDocValuesGeoPointsTests Completed [238/1066] on J0 in 0.14s, 2 tests Suite: org.elasticsearch.search.aggregations.metrics.percentiles.tdigest.TDigestPercentilesAggregatorTests Completed [239/1066] on J2 in 0.36s, 5 tests Suite: org.elasticsearch.action.main.MainResponseTests Completed [240/1066] on J2 in 0.15s, 4 tests Suite: org.elasticsearch.action.update.UpdateResponseTests Completed [241/1066] on J0 in 0.30s, 3 tests Suite: org.elasticsearch.index.query.SpanContainingQueryBuilderTests Completed [242/1066] on J0 in 0.38s, 11 tests Suite: org.elasticsearch.index.query.RewriteableTests Completed [243/1066] on J2 in 0.73s, 3 tests Suite: org.elasticsearch.action.admin.indices.template.put.MetaDataIndexTemplateServiceTests Completed [244/1066] on J1 in 1.26s, 9 tests Suite: org.elasticsearch.action.search.TransportSearchHelperTests Completed [245/1066] on J1 in 0.06s, 1 test Suite: org.elasticsearch.index.IndexSortSettingsTests Completed [246/1066] on J1 in 0.06s, 9 tests Suite: org.elasticsearch.search.aggregations.bucket.sampler.BestDocsDeferringCollectorTests Completed [247/1066] on J1 in 0.06s, 1 test Suite: org.elasticsearch.index.analysis.ShingleTokenFilterFactoryTests Completed [248/1066] on J0 in 0.11s, 6 tests Suite: org.elasticsearch.cluster.metadata.AliasValidatorTests Completed [249/1066] on J0 in 0.02s, 1 test Suite: org.elasticsearch.cluster.NodeConnectionsServiceTests Completed [250/1066] on J1 in 0.09s, 2 tests Suite: org.elasticsearch.script.ClassPermissionTests Completed [251/1066] on J1 in 0.04s, 7 tests Suite: org.elasticsearch.monitor.jvm.JvmMonitorTests Completed [252/1066] on J1 in 0.07s, 4 tests Suite: org.elasticsearch.action.support.master.TransportMasterNodeActionTests Completed [253/1066] on J1 in 0.10s, 9 tests Suite: org.elasticsearch.cluster.routing.allocation.RandomAllocationDeciderTests Completed [254/1066] on J1 in 0.08s, 1 test Suite: org.elasticsearch.search.aggregations.bucket.histogram.ExtendedBoundsTests Completed [255/1066] on J1 in 0.08s, 3 tests Suite: org.elasticsearch.action.search.SearchResponseTests Completed [256/1066] on J1 in 0.29s, 6 tests Suite: org.elasticsearch.bootstrap.SecurityTests Completed [257/1066] on J1 in 0.23s, 4 tests Suite: org.elasticsearch.common.rounding.TimeZoneRoundingTests Completed [258/1066] on J2 in 1.51s, 22 tests Suite: org.elasticsearch.index.shard.ShardPathTests IGNOR/A 0.02s J1 | ShardPathTests.testFailLoadShardPathOnMultiState > Assumption #1: This test tests multi data.path but we only got one Completed [259/1066] on J1 in 0.35s, 6 tests, 1 skipped Suite: org.elasticsearch.index.mapper.IpFieldTypeTests Completed [260/1066] on J2 in 0.11s, 10 tests Suite: org.elasticsearch.common.io.stream.BytesStreamsTests Completed [261/1066] on J1 in 0.74s, 37 tests Suite: org.elasticsearch.threadpool.ScheduleWithFixedDelayTests Completed [262/1066] on J2 in 0.67s, 7 tests Suite: org.elasticsearch.common.xcontent.XContentFactoryTests Completed [263/1066] on J1 in 0.05s, 9 tests Suite: org.elasticsearch.search.aggregations.metrics.percentiles.tdigest.TDigestPercentileRanksAggregatorTests Completed [264/1066] on J1 in 0.06s, 4 tests Suite: org.elasticsearch.action.main.MainActionTests Completed [265/1066] on J1 in 0.06s, 1 test Suite: org.elasticsearch.index.query.RegexpQueryBuilderTests IGNOR/A 0.01s J1 | RegexpQueryBuilderTests.testNumeric > Assumption #1: test runs only when at least a type is registered Completed [266/1066] on J1 in 0.15s, 13 tests, 1 skipped Suite: org.elasticsearch.common.rounding.DateTimeUnitTests Completed [267/1066] on J1 in 0.03s, 1 test Suite: org.elasticsearch.index.mapper.BinaryFieldMapperTests Completed [268/1066] on J2 in 0.54s, 3 tests Suite: org.elasticsearch.common.lucene.search.QueriesTests Completed [269/1066] on J2 in 0.10s, 3 tests Suite: org.elasticsearch.cluster.routing.ShardRoutingTests Completed [270/1066] on J2 in 0.03s, 4 tests Suite: org.elasticsearch.cluster.metadata.ToAndFromJsonMetaDataTests Completed [271/1066] on J2 in 0.04s, 1 test Suite: org.elasticsearch.index.mapper.IpFieldMapperTests Completed [272/1066] on J1 in 0.97s, 8 tests Suite: org.elasticsearch.threadpool.ScalingThreadPoolTests Completed [273/1066] on J1 in 0.15s, 3 tests Suite: org.elasticsearch.index.mapper.AllFieldMapperTests Completed [274/1066] on J1 in 0.21s, 1 test Suite: org.elasticsearch.common.lucene.search.MultiPhrasePrefixQueryTests Completed [275/1066] on J1 in 0.04s, 1 test Suite: org.elasticsearch.cluster.routing.RoutingTableTests Completed [276/1066] on J1 in 0.10s, 12 tests Suite: org.elasticsearch.cluster.metadata.TemplateUpgradeServiceTests Completed [277/1066] on J1 in 0.09s, 3 tests Suite: org.elasticsearch.transport.TransportActionProxyTests Completed [278/1066] on J2 in 1.19s, 6 tests Suite: org.elasticsearch.transport.TcpTransportTests Completed [279/1066] on J1 in 0.08s, 22 tests Suite: org.elasticsearch.search.suggest.completion.FuzzyOptionsTests Completed [280/1066] on J1 in 0.04s, 3 tests Suite: org.elasticsearch.rest.action.cat.RestTableTests Completed [281/1066] on J1 in 0.07s, 15 tests Suite: org.elasticsearch.index.analysis.SnowballAnalyzerTests Completed [282/1066] on J1 in 0.15s, 4 tests Suite: org.elasticsearch.search.suggest.phrase.LinearInterpolationModelTests Completed [283/1066] on J1 in 0.09s, 4 tests Suite: org.elasticsearch.common.FieldMemoryStatsTests Completed [284/1066] on J1 in 0.08s, 3 tests Suite: org.elasticsearch.search.internal.ShardSearchTransportRequestTests Completed [285/1066] on J1 in 0.04s, 4 tests Suite: org.elasticsearch.cluster.routing.AllocationIdTests Completed [286/1066] on J1 in 0.03s, 6 tests Suite: org.elasticsearch.search.suggest.completion.GeoContextMappingTests Completed [287/1066] on J2 in 0.74s, 9 tests Suite: org.elasticsearch.rest.action.document.RestBulkActionTests Completed [288/1066] on J2 in 0.05s, 1 test Suite: org.elasticsearch.action.admin.indices.rollover.RolloverResponseTests Completed [289/1066] on J1 in 0.05s, 3 tests Suite: org.elasticsearch.search.aggregations.bucket.significant.SignificantTextAggregatorTests Completed [290/1066] on J1 in 0.20s, 2 tests Suite: org.elasticsearch.indices.recovery.StartRecoveryRequestTests Completed [291/1066] on J1 in 0.03s, 1 test Suite: org.elasticsearch.index.analysis.StopAnalyzerTests Completed [292/1066] on J2 in 0.04s, 1 test Suite: org.elasticsearch.index.search.ESToParentBlockJoinQueryTests Completed [293/1066] on J1 in 0.09s, 2 tests Suite: org.elasticsearch.search.suggest.phrase.NoisyChannelSpellCheckerTests Completed [294/1066] on J2 in 0.41s, 4 tests Suite: org.elasticsearch.common.NumbersTests Completed [295/1066] on J2 in 0.03s, 5 tests Suite: org.elasticsearch.search.profile.ProfileResultTests Completed [296/1066] on J2 in 0.03s, 3 tests Suite: org.elasticsearch.index.query.CommonTermsQueryParserTests Completed [297/1066] on J1 in 0.48s, 1 test Suite: org.elasticsearch.common.TableTests Completed [298/1066] on J1 in 0.03s, 11 tests Suite: org.elasticsearch.cluster.routing.DelayedAllocationServiceTests Completed [299/1066] on J2 in 1.05s, 4 tests Suite: org.elasticsearch.action.admin.indices.rollover.TransportRolloverActionTests Completed [300/1066] on J2 in 0.13s, 8 tests Suite: org.elasticsearch.search.aggregations.bucket.terms.DoubleTermsTests Completed [301/1066] on J2 in 0.32s, 6 tests Suite: org.elasticsearch.indices.store.IndicesStoreTests Completed [302/1066] on J2 in 0.07s, 3 tests Suite: org.elasticsearch.index.search.MultiMatchQueryTests Completed [303/1066] on J2 in 1.78s, 8 tests Suite: org.elasticsearch.index.query.ConstantScoreQueryBuilderTests IGNOR/A 0.00s J2 | ConstantScoreQueryBuilderTests.testMultipleFilterElements > Assumption #1: Test only makes sense if XContent parser doesn't have strict duplicate checks enabled IGNOR/A 0.00s J2 | ConstantScoreQueryBuilderTests.testUnknownField > Assumption #1: test doesn't apply for query filter queries Completed [304/1066] on J2 in 0.12s, 15 tests, 2 skipped Suite: org.elasticsearch.index.translog.TranslogTests Completed [305/1066] on J1 in 7.11s, 54 tests Suite: org.elasticsearch.action.index.IndexRequestTests Completed [306/1066] on J1 in 0.07s, 10 tests Suite: org.elasticsearch.search.aggregations.pipeline.derivative.InternalDerivativeTests Completed [307/1066] on J1 in 0.13s, 5 tests Suite: org.elasticsearch.common.UUIDTests Completed [308/1066] on J2 in 4.62s, 5 tests Suite: org.elasticsearch.index.store.DirectoryUtilsTests Completed [309/1066] on J1 in 0.42s, 1 test Suite: org.elasticsearch.common.util.concurrent.RefCountedTests Completed [310/1066] on J1 in 0.12s, 2 tests Suite: org.elasticsearch.ingest.ValueSourceTests Completed [311/1066] on J1 in 0.07s, 3 tests Suite: org.elasticsearch.search.aggregations.pipeline.bucketmetrics.percentile.InternalPercentilesBucketTests Completed [312/1066] on J1 in 0.06s, 8 tests Suite: org.elasticsearch.discovery.zen.NodeRemovalClusterStateTaskExecutorTests Completed [313/1066] on J1 in 0.05s, 3 tests Suite: org.elasticsearch.search.aggregations.metrics.percentiles.hdr.InternalHDRPercentilesTests Completed [314/1066] on J1 in 0.38s, 7 tests Suite: org.elasticsearch.rest.action.admin.indices.RestClearIndicesCacheActionTests Completed [315/1066] on J1 in 0.03s, 1 test Suite: org.elasticsearch.ingest.ConfigurationUtilsTests Completed [316/1066] on J1 in 0.08s, 10 tests Suite: org.elasticsearch.action.delete.DeleteResponseTests Completed [317/1066] on J1 in 0.06s, 3 tests Suite: org.elasticsearch.index.analysis.FingerprintAnalyzerTests Completed [318/1066] on J1 in 0.16s, 4 tests Suite: org.elasticsearch.search.rescore.QueryRescoreModeTests Completed [319/1066] on J1 in 0.28s, 1 test Suite: org.elasticsearch.transport.RemoteClusterServiceTests Completed [320/1066] on J0 in 14.97s, 9 tests Suite: org.elasticsearch.action.admin.indices.open.OpenIndexResponseTests Completed [321/1066] on J0 in 0.11s, 3 tests Suite: org.elasticsearch.cluster.metadata.MetaDataTests Completed [322/1066] on J0 in 0.17s, 11 tests Suite: org.elasticsearch.common.geo.GeoWKTShapeParserTests Completed [323/1066] on J1 in 0.46s, 16 tests Suite: org.elasticsearch.action.admin.cluster.settings.ClusterUpdateSettingsResponseTests Completed [324/1066] on J1 in 0.17s, 3 tests Suite: org.elasticsearch.cluster.routing.RoutingServiceTests Completed [325/1066] on J0 in 0.16s, 1 test Suite: org.elasticsearch.action.search.ClearScrollRequestTests Completed [326/1066] on J1 in 0.12s, 5 tests Suite: org.elasticsearch.common.util.iterable.IterablesTests Completed [327/1066] on J1 in 0.10s, 3 tests Suite: org.elasticsearch.common.lucene.ShardCoreKeyMapTests Completed [328/1066] on J1 in 0.25s, 3 tests Suite: org.elasticsearch.action.termvectors.TermVectorsUnitTests Completed [329/1066] on J1 in 0.24s, 9 tests Suite: org.elasticsearch.search.aggregations.bucket.range.BinaryRangeAggregatorTests Completed [330/1066] on J1 in 0.14s, 4 tests Suite: org.elasticsearch.index.reindex.LeaderBulkByScrollTaskStateTests Completed [331/1066] on J1 in 0.20s, 2 tests Suite: org.elasticsearch.indices.cluster.IndicesClusterStateServiceRandomUpdatesTests Completed [332/1066] on J0 in 1.30s, 2 tests Suite: org.elasticsearch.indices.IndexingMemoryControllerTests Completed [333/1066] on J2 in 3.28s, 11 tests Suite: org.elasticsearch.action.index.IndexResponseTests Completed [334/1066] on J2 in 0.03s, 3 tests Suite: org.elasticsearch.search.aggregations.pipeline.moving.avg.MovAvgTests Completed [335/1066] on J2 in 0.05s, 4 tests Suite: org.elasticsearch.cluster.metadata.IndexMetaDataTests Completed [336/1066] on J1 in 0.07s, 7 tests Suite: org.elasticsearch.search.aggregations.pipeline.bucketmetrics.MinBucketTests Completed [337/1066] on J0 in 0.11s, 4 tests Suite: org.elasticsearch.common.ExponentiallyWeightedMovingAverageTests Completed [338/1066] on J0 in 0.04s, 3 tests Suite: org.elasticsearch.discovery.single.SingleNodeDiscoveryTests Completed [339/1066] on J0 in 0.09s, 1 test Suite: org.elasticsearch.index.store.FsDirectoryServiceTests Completed [340/1066] on J2 in 0.05s, 1 test Suite: org.elasticsearch.plugins.PluginInfoTests Completed [341/1066] on J1 in 0.23s, 15 tests Suite: org.elasticsearch.common.util.concurrent.ReleasableLockTests Completed [342/1066] on J2 in 0.02s, 1 test Suite: org.elasticsearch.monitor.fs.DeviceStatsTests Completed [343/1066] on J2 in 0.05s, 1 test Suite: org.elasticsearch.search.aggregations.pipeline.bucketmetrics.stats.extended.InternalExtendedStatsBucketTests Completed [344/1066] on J2 in 0.08s, 6 tests Suite: org.elasticsearch.search.sort.FieldSortBuilderTests Completed [345/1066] on J0 in 0.14s, 13 tests Suite: org.elasticsearch.discovery.zen.PendingClusterStatesQueueTests Completed [346/1066] on J2 in 0.12s, 7 tests Suite: org.elasticsearch.search.aggregations.metrics.percentiles.tdigest.InternalTDigestPercentilesRanksTests Completed [347/1066] on J2 in 0.13s, 6 tests Suite: org.elasticsearch.cluster.metadata.DateMathExpressionResolverTests Completed [348/1066] on J0 in 0.25s, 15 tests Suite: org.elasticsearch.threadpool.ThreadPoolSerializationTests Completed [349/1066] on J0 in 0.07s, 6 tests Suite: org.elasticsearch.common.settings.ListKeyStoreCommandTests Completed [350/1066] on J1 in 0.55s, 4 tests Suite: org.elasticsearch.cluster.routing.allocation.ShardVersioningTests Completed [351/1066] on J1 in 0.05s, 1 test Suite: org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequestTests Completed [352/1066] on J1 in 0.14s, 6 tests Suite: org.elasticsearch.rest.action.admin.indices.RestIndicesStatsActionTests Completed [353/1066] on J2 in 0.05s, 3 tests Suite: org.elasticsearch.common.geo.builders.PointBuilderTests Completed [354/1066] on J0 in 0.31s, 3 tests Suite: org.elasticsearch.index.mapper.BinaryRangeUtilTests Completed [355/1066] on J1 in 0.45s, 5 tests Suite: org.elasticsearch.ingest.IngestDocumentTests Completed [356/1066] on J2 in 0.40s, 87 tests Suite: org.elasticsearch.action.fieldcaps.FieldCapabilitiesRequestTests Completed [357/1066] on J2 in 0.25s, 3 tests Suite: org.elasticsearch.script.ScriptMetaDataTests Completed [358/1066] on J1 in 0.58s, 7 tests Suite: org.elasticsearch.index.fielddata.FieldDataStatsTests Completed [359/1066] on J1 in 0.15s, 1 test Suite: org.elasticsearch.index.analysis.HunspellTokenFilterFactoryTests Completed [360/1066] on J2 in 2.43s, 1 test Suite: org.elasticsearch.index.analysis.PatternAnalyzerTests Completed [361/1066] on J1 in 2.10s, 6 tests Suite: org.elasticsearch.search.rescore.QueryRescorerBuilderTests Completed [362/1066] on J2 in 0.54s, 7 tests Suite: org.elasticsearch.search.aggregations.metrics.TopHitsTests Completed [363/1066] on J1 in 0.31s, 6 tests Suite: org.elasticsearch.common.geo.ShapeBuilderTests Completed [364/1066] on J2 in 0.47s, 26 tests Suite: org.elasticsearch.action.admin.cluster.settings.SettingsUpdaterTests Completed [365/1066] on J2 in 0.41s, 6 tests Suite: org.elasticsearch.script.StoredScriptSourceTests Completed [366/1066] on J1 in 0.74s, 3 tests Suite: org.elasticsearch.action.search.CountedCollectorTests Completed [367/1066] on J2 in 0.12s, 1 test Suite: org.elasticsearch.action.admin.cluster.reroute.ClusterRerouteResponseTests Completed [368/1066] on J1 in 0.06s, 1 test Suite: org.elasticsearch.search.aggregations.pipeline.SerialDifferenceTests Completed [369/1066] on J1 in 0.22s, 3 tests Suite: org.elasticsearch.common.io.stream.StreamTests Completed [370/1066] on J1 in 0.07s, 9 tests Suite: org.elasticsearch.search.suggest.completion.RegexOptionsTests Completed [371/1066] on J1 in 0.06s, 2 tests Suite: org.elasticsearch.common.util.set.SetsTests Completed [372/1066] on J2 in 0.18s, 2 tests Suite: org.elasticsearch.common.lucene.index.ESDirectoryReaderTests Completed [373/1066] on J2 in 0.16s, 1 test Suite: org.elasticsearch.index.translog.MultiSnapshotTests Completed [374/1066] on J1 in 0.19s, 4 tests Suite: org.elasticsearch.action.update.UpdateRequestTests Completed [375/1066] on J2 in 0.27s, 15 tests Suite: org.elasticsearch.index.query.SpanFirstQueryBuilderTests Completed [376/1066] on J1 in 0.24s, 11 tests Suite: org.elasticsearch.common.lucene.search.morelikethis.MoreLikeThisQueryTests Completed [377/1066] on J1 in 0.05s, 1 test Suite: org.elasticsearch.search.aggregations.bucket.range.InternalBinaryRangeTests Completed [378/1066] on J2 in 0.22s, 6 tests Suite: org.elasticsearch.action.admin.indices.shards.IndicesShardStoreResponseTests Completed [379/1066] on J1 in 0.04s, 2 tests Suite: org.elasticsearch.index.reindex.ReindexRequestTests Completed [380/1066] on J2 in 0.06s, 4 tests Suite: org.elasticsearch.common.geo.builders.MultiPointBuilderTests Completed [381/1066] on J1 in 0.28s, 4 tests Suite: org.elasticsearch.cluster.metadata.IndexNameExpressionResolverTests Completed [382/1066] on J2 in 0.49s, 44 tests Suite: org.elasticsearch.cluster.routing.allocation.FailedShardsRoutingTests Completed [383/1066] on J1 in 0.17s, 9 tests Suite: org.elasticsearch.action.ingest.SimulateDocumentSimpleResultTests Completed [384/1066] on J1 in 0.06s, 1 test Suite: org.elasticsearch.index.query.SpanNearQueryBuilderTests Completed [385/1066] on J1 in 0.27s, 14 tests Suite: org.elasticsearch.common.path.PathTrieTests Completed [386/1066] on J1 in 0.06s, 10 tests Suite: org.elasticsearch.common.cache.CacheTests Completed [387/1066] on J0 in 6.99s, 24 tests Suite: org.elasticsearch.index.query.functionscore.FunctionScoreTests Completed [388/1066] on J0 in 0.20s, 11 tests Suite: org.elasticsearch.common.geo.GeoUtilTests Completed [389/1066] on J0 in 0.04s, 2 tests Suite: org.elasticsearch.ExceptionsHelperTests Completed [390/1066] on J0 in 1.25s, 2 tests Suite: org.elasticsearch.index.store.IndexStoreTests Completed [391/1066] on J0 in 0.10s, 1 test Suite: org.elasticsearch.snapshots.RepositoriesMetaDataSerializationTests Completed [392/1066] on J0 in 0.27s, 4 tests Suite: org.elasticsearch.plugins.PluginsServiceTests IGNOR/A 0.00s J2 | PluginsServiceTests.testExistingMandatoryMetaPlugin > Assumption #1: windows deletion behavior is asinine IGNOR/A 0.00s J2 | PluginsServiceTests.testNonExtensibleDep > Assumption #1: windows deletion behavior is asinine IGNOR/A 0.00s J2 | PluginsServiceTests.testExistingMandatoryInstalledPlugin > Assumption #1: windows deletion behavior is asinine Completed [393/1066] on J2 in 3.14s, 31 tests, 3 skipped Suite: org.elasticsearch.cluster.routing.allocation.DeadNodesAllocationTests Completed [394/1066] on J0 in 0.17s, 3 tests Suite: org.elasticsearch.common.settings.MemorySizeSettingsTests Completed [395/1066] on J2 in 0.07s, 6 tests Suite: org.elasticsearch.cluster.routing.allocation.ShardsLimitAllocationTests Completed [396/1066] on J2 in 0.28s, 3 tests Suite: org.elasticsearch.action.admin.indices.mapping.put.PutMappingResponseTests Completed [397/1066] on J2 in 0.07s, 4 tests Suite: org.elasticsearch.common.xcontent.support.XContentMapValuesTests Completed [398/1066] on J0 in 0.59s, 51 tests Suite: org.elasticsearch.index.query.BoolQueryBuilderTests IGNOR/A 0.06s J0 | BoolQueryBuilderTests.testTooManyQueriesInObject > Assumption #1: Test only makes sense if XContent parser doesn't have strict duplicate checks enabled Completed [399/1066] on J0 in 0.55s, 22 tests, 1 skipped Suite: org.elasticsearch.cluster.routing.allocation.FailedNodeRoutingTests Completed [400/1066] on J0 in 0.26s, 2 tests Suite: org.elasticsearch.search.aggregations.pipeline.bucketmetrics.SumBucketTests Completed [401/1066] on J0 in 0.05s, 4 tests Suite: org.elasticsearch.common.lucene.search.function.MinScoreScorerTests Completed [402/1066] on J0 in 0.06s, 2 tests Suite: org.elasticsearch.index.mapper.BooleanFieldMapperTests Completed [403/1066] on J2 in 1.38s, 6 tests Suite: org.elasticsearch.script.ScriptServiceTests Completed [404/1066] on J2 in 0.20s, 19 tests Suite: org.elasticsearch.index.fielddata.FieldDataTests Completed [405/1066] on J2 in 0.07s, 2 tests Suite: org.elasticsearch.index.mapper.DynamicTemplatesTests Completed [406/1066] on J0 in 0.72s, 3 tests Suite: org.elasticsearch.index.analysis.PreBuiltAnalyzerTests Completed [407/1066] on J2 in 0.45s, 4 tests Suite: org.elasticsearch.search.aggregations.metrics.ValueCountTests Completed [408/1066] on J2 in 0.12s, 5 tests Suite: org.elasticsearch.index.mapper.GenericStoreDynamicTemplateTests Completed [409/1066] on J0 in 0.42s, 1 test Suite: org.elasticsearch.script.StoredScriptTests Completed [410/1066] on J2 in 0.17s, 7 tests Suite: org.elasticsearch.action.admin.cluster.reroute.ClusterRerouteTests Completed [411/1066] on J2 in 0.06s, 2 tests Suite: org.elasticsearch.search.aggregations.pipeline.bucketmetrics.AvgBucketTests Completed [412/1066] on J2 in 0.05s, 4 tests Suite: org.elasticsearch.index.engine.InternalEngineSettingsTests Completed [413/1066] on J0 in 0.44s, 1 test Suite: org.elasticsearch.common.joda.DateMathParserTests Completed [414/1066] on J2 in 0.10s, 15 tests Suite: org.elasticsearch.search.suggest.phrase.DirectCandidateGeneratorTests Completed [415/1066] on J2 in 0.08s, 5 tests Suite: org.elasticsearch.ingest.PipelineConfigurationTests Completed [416/1066] on J0 in 0.04s, 2 tests Suite: org.elasticsearch.repositories.IndexIdTests Completed [417/1066] on J0 in 0.04s, 3 tests Suite: org.elasticsearch.index.query.CommonTermsQueryBuilderTests Completed [418/1066] on J0 in 0.19s, 16 tests Suite: org.elasticsearch.index.translog.TranslogDeletionPolicyTests Completed [419/1066] on J2 in 0.31s, 4 tests Suite: org.elasticsearch.ingest.IngestMetadataTests Completed [420/1066] on J0 in 0.05s, 2 tests Suite: org.elasticsearch.search.ClearScrollResponseTests Completed [421/1066] on J0 in 0.03s, 2 tests Suite: org.elasticsearch.search.scroll.RestClearScrollActionTests Completed [422/1066] on J0 in 0.12s, 2 tests Suite: org.elasticsearch.index.query.SpanGapQueryBuilderTests Completed [423/1066] on J2 in 0.18s, 11 tests Suite: org.elasticsearch.common.lucene.search.morelikethis.XMoreLikeThisTests Completed [424/1066] on J2 in 0.44s, 1 test Suite: org.elasticsearch.action.admin.indices.shrink.ResizeRequestTests Completed [425/1066] on J2 in 0.18s, 2 tests Suite: org.elasticsearch.search.aggregations.metrics.tophits.TopHitsAggregatorTests Completed [426/1066] on J0 in 1.23s, 4 tests Suite: org.elasticsearch.index.shard.ShardUtilsTests Completed [427/1066] on J0 in 0.06s, 1 test Suite: org.elasticsearch.common.geo.builders.MultiPolygonBuilderTests Completed [428/1066] on J2 in 0.49s, 3 tests Suite: org.elasticsearch.action.bulk.BulkShardRequestTests Completed [429/1066] on J0 in 0.06s, 1 test Suite: org.elasticsearch.common.geo.builders.MultiLineStringBuilderTests Completed [430/1066] on J0 in 0.14s, 3 tests Suite: org.elasticsearch.cluster.routing.allocation.FilterRoutingTests Completed [431/1066] on J2 in 0.19s, 3 tests Suite: org.elasticsearch.cluster.routing.PlainShardIteratorTests Completed [432/1066] on J0 in 0.02s, 1 test Suite: org.elasticsearch.action.ingest.SimulateExecutionServiceTests Completed [433/1066] on J2 in 0.18s, 7 tests Suite: org.apache.lucene.search.uhighlight.BoundedBreakIteratorScannerTests Completed [434/1066] on J0 in 0.31s, 1 test Suite: org.elasticsearch.index.query.SpanNotQueryBuilderTests Completed [435/1066] on J2 in 0.17s, 15 tests Suite: org.elasticsearch.common.recycler.ConcurrentRecyclerTests Completed [436/1066] on J2 in 0.03s, 5 tests Suite: org.elasticsearch.gateway.MetaStateServiceTests Completed [437/1066] on J0 in 0.30s, 5 tests Suite: org.elasticsearch.common.hashing.MurmurHash3Tests Completed [438/1066] on J0 in 0.03s, 1 test Suite: org.elasticsearch.index.mapper.DoubleIndexingDocTests Completed [439/1066] on J2 in 0.39s, 1 test Suite: org.elasticsearch.search.suggest.term.TermSuggestionBuilderTests Completed [440/1066] on J2 in 0.14s, 7 tests Suite: org.elasticsearch.index.suggest.stats.CompletionsStatsTests Completed [441/1066] on J2 in 0.03s, 1 test Suite: org.elasticsearch.action.bulk.TransportBulkActionTookTests Completed [442/1066] on J2 in 0.04s, 2 tests Suite: org.elasticsearch.search.aggregations.bucket.missing.InternalMissingTests Completed [443/1066] on J2 in 0.10s, 5 tests Suite: org.elasticsearch.VersionTests Completed [444/1066] on J2 in 0.09s, 24 tests Suite: org.elasticsearch.common.logging.LoggersTests Completed [445/1066] on J2 in 0.03s, 1 test Suite: org.elasticsearch.search.aggregations.bucket.sampler.SamplerAggregatorTests Completed [446/1066] on J2 in 0.06s, 1 test Suite: org.elasticsearch.index.similarity.ScriptedSimilarityTests Completed [447/1066] on J2 in 0.10s, 4 tests Suite: org.elasticsearch.common.recycler.QueueRecyclerTests Completed [448/1066] on J2 in 0.06s, 5 tests Suite: org.elasticsearch.index.mapper.DocumentParserTests Completed [449/1066] on J1 in 9.51s, 70 tests Suite: org.elasticsearch.search.suggest.term.SuggestModeTests Completed [450/1066] on J1 in 0.08s, 4 tests Suite: org.elasticsearch.index.store.StoreTests Completed [451/1066] on J1 in 0.47s, 21 tests Suite: org.elasticsearch.index.search.nested.FloatNestedSortingTests Completed [452/1066] on J2 in 0.80s, 2 tests Suite: org.elasticsearch.rest.action.admin.cluster.RestNodesStatsActionTests Completed [453/1066] on J2 in 0.08s, 7 tests Suite: org.elasticsearch.action.bulk.TransportBulkActionTests Completed [454/1066] on J1 in 0.05s, 3 tests Suite: org.elasticsearch.search.aggregations.bucket.histogram.InternalHistogramTests Completed [455/1066] on J1 in 0.22s, 7 tests Suite: org.elasticsearch.index.fielddata.ScriptDocValuesDatesTests Completed [456/1066] on J2 in 0.27s, 1 test Suite: org.elasticsearch.SpecialPermissionTests Completed [457/1066] on J1 in 0.18s, 2 tests Suite: org.elasticsearch.common.logging.DeprecationLoggerTests Completed [458/1066] on J1 in 0.16s, 15 tests Suite: org.elasticsearch.search.aggregations.metrics.percentiles.hdr.HDRPercentilesAggregatorTests Completed [459/1066] on J2 in 0.20s, 5 tests Suite: org.elasticsearch.search.aggregations.bucket.sampler.InternalSamplerTests Completed [460/1066] on J1 in 0.07s, 5 tests Suite: org.elasticsearch.bootstrap.ElasticsearchCliTests Completed [461/1066] on J2 in 0.40s, 9 tests Suite: org.elasticsearch.bootstrap.JavaVersionTests Completed [462/1066] on J2 in 0.22s, 6 tests Suite: org.elasticsearch.index.shard.StoreRecoveryTests Completed [463/1066] on J1 in 0.64s, 3 tests Suite: org.elasticsearch.common.settings.KeyStoreWrapperTests Completed [464/1066] on J0 in 3.17s, 11 tests Suite: org.elasticsearch.common.recycler.NoneRecyclerTests Completed [465/1066] on J1 in 0.08s, 5 tests Suite: org.elasticsearch.search.suggest.completion.GeoQueryContextTests Completed [466/1066] on J0 in 0.08s, 4 tests Suite: org.elasticsearch.ingest.CompoundProcessorTests Completed [467/1066] on J0 in 0.08s, 10 tests Suite: org.elasticsearch.common.settings.PropertyPlaceholderTests Completed [468/1066] on J0 in 0.05s, 12 tests Suite: org.elasticsearch.index.mapper.RoutingFieldTypeTests Completed [469/1066] on J0 in 0.07s, 5 tests Suite: org.elasticsearch.search.aggregations.metrics.MaxTests Completed [470/1066] on J0 in 0.14s, 5 tests Suite: org.elasticsearch.index.search.nested.DoubleNestedSortingTests Completed [471/1066] on J1 in 0.96s, 2 tests Suite: org.elasticsearch.common.xcontent.cbor.CborXContentTests Completed [472/1066] on J0 in 0.37s, 50 tests Suite: org.elasticsearch.rest.action.RestMainActionTests Completed [473/1066] on J1 in 0.15s, 2 tests Suite: org.elasticsearch.index.mapper.NestedObjectMapperTests Completed [474/1066] on J2 in 3.94s, 14 tests Suite: org.elasticsearch.cluster.routing.allocation.decider.RestoreInProgressAllocationDeciderTests Completed [475/1066] on J2 in 0.06s, 3 tests Suite: org.elasticsearch.cluster.routing.allocation.RoutingNodesIntegrityTests Completed [476/1066] on J2 in 0.18s, 3 tests Suite: org.elasticsearch.common.xcontent.json.JsonXContentTests Completed [477/1066] on J2 in 0.39s, 50 tests Suite: org.elasticsearch.threadpool.ThreadPoolTests Completed [478/1066] on J2 in 0.11s, 4 tests Suite: org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitorTests Completed [479/1066] on J2 in 0.05s, 1 test Suite: org.elasticsearch.index.query.AbstractQueryBuilderTests Completed [480/1066] on J2 in 0.04s, 2 tests Suite: org.elasticsearch.common.transport.BoundTransportAddressTests Completed [481/1066] on J2 in 0.06s, 2 tests Suite: org.elasticsearch.index.replication.RecoveryDuringReplicationTests Completed [482/1066] on J0 in 6.56s, 8 tests Suite: org.elasticsearch.common.collect.IteratorsTests Completed [483/1066] on J0 in 0.12s, 12 tests Suite: org.elasticsearch.rest.action.RestBuilderListenerTests Completed [484/1066] on J0 in 0.03s, 3 tests Suite: org.elasticsearch.index.fielddata.PagedBytesStringFieldDataTests Completed [485/1066] on J1 in 6.38s, 19 tests Suite: org.elasticsearch.search.aggregations.metrics.percentiles.hdr.HDRPercentileRanksAggregatorTests Completed [486/1066] on J1 in 0.06s, 4 tests Suite: org.elasticsearch.bootstrap.ESPolicyTests Completed [487/1066] on J1 in 0.04s, 1 test Suite: org.elasticsearch.bootstrap.JNANativesTests Completed [488/1066] on J1 in 0.02s, 2 tests Suite: org.elasticsearch.index.mapper.CompletionFieldMapperTests Completed [489/1066] on J2 in 3.27s, 16 tests Suite: org.elasticsearch.cluster.routing.allocation.StartedShardsRoutingTests Completed [490/1066] on J2 in 0.04s, 2 tests Suite: org.elasticsearch.cluster.routing.allocation.MoveDecisionTests Completed [491/1066] on J0 in 0.08s, 4 tests Suite: org.elasticsearch.cluster.action.shard.ShardFailedClusterStateTaskExecutorTests Completed [492/1066] on J0 in 0.12s, 6 tests Suite: org.elasticsearch.action.OriginalIndicesTests Completed [493/1066] on J0 in 0.04s, 1 test Suite: org.elasticsearch.search.aggregations.metrics.geocentroid.InternalGeoCentroidTests Completed [494/1066] on J2 in 0.08s, 5 tests Suite: org.elasticsearch.common.xcontent.support.filtering.FilterPathGeneratorFilteringTests Completed [495/1066] on J2 in 0.06s, 4 tests Suite: org.elasticsearch.common.unit.SizeValueTests Completed [496/1066] on J2 in 0.06s, 7 tests Suite: org.elasticsearch.index.search.geo.GeoUtilsTests Completed [497/1066] on J0 in 0.22s, 38 tests Suite: org.elasticsearch.index.mapper.TextFieldTypeTests Completed [498/1066] on J2 in 0.08s, 9 tests Suite: org.elasticsearch.search.suggest.term.SortByTests Completed [499/1066] on J2 in 0.08s, 4 tests Suite: org.elasticsearch.index.mapper.StoredNumericValuesTests Completed [500/1066] on J0 in 0.42s, 1 test Suite: org.elasticsearch.common.util.concurrent.CountDownTests Completed [501/1066] on J0 in 0.08s, 2 tests Suite: org.elasticsearch.search.aggregations.bucket.geogrid.GeoHashGridParserTests Completed [502/1066] on J0 in 0.04s, 7 tests Suite: org.elasticsearch.index.mapper.MultiFieldTests Completed [503/1066] on J1 in 1.00s, 6 tests Suite: org.elasticsearch.cluster.routing.allocation.decider.FilterAllocationDeciderTests Completed [504/1066] on J1 in 0.05s, 4 tests Suite: org.elasticsearch.index.snapshots.blobstore.SlicedInputStreamTests Completed [505/1066] on J0 in 0.04s, 1 test Suite: org.elasticsearch.cluster.routing.allocation.ResizeAllocationDeciderTests Completed [506/1066] on J1 in 0.07s, 5 tests Suite: org.elasticsearch.watcher.ResourceWatcherServiceTests Completed [507/1066] on J0 in 0.03s, 2 tests Suite: org.elasticsearch.action.admin.cluster.shards.ClusterSearchShardsRequestTests Completed [508/1066] on J0 in 0.03s, 2 tests Suite: org.elasticsearch.cluster.routing.allocation.DiskThresholdSettingsTests Completed [509/1066] on J0 in 0.11s, 9 tests Suite: org.elasticsearch.common.xcontent.cbor.JsonVsCborTests Completed [510/1066] on J1 in 0.02s, 1 test Suite: org.elasticsearch.threadpool.ThreadPoolStatsTests Completed [511/1066] on J1 in 0.03s, 2 tests Suite: org.elasticsearch.cluster.routing.allocation.DecisionsImpactOnClusterHealthTests Completed [512/1066] on J1 in 0.04s, 3 tests Suite: org.elasticsearch.index.mapper.VersionFieldTypeTests Completed [513/1066] on J1 in 0.07s, 5 tests Suite: org.elasticsearch.common.settings.SettingsTests IGNOR/A 0.02s J1 | SettingsTests.testDuplicateKeysThrowsException > Assumption #1: Test only makes sense if XContent parser doesn't have strict duplicate checks enabled Completed [514/1066] on J1 in 0.28s, 33 tests, 1 skipped Suite: org.elasticsearch.search.aggregations.metrics.cardinality.HyperLogLogPlusPlusTests Completed [515/1066] on J0 in 1.28s, 5 tests Suite: org.apache.lucene.queries.BinaryDocValuesRangeQueryTests Completed [516/1066] on J0 in 0.40s, 2 tests Suite: org.elasticsearch.common.geo.ShapeRelationTests Completed [517/1066] on J0 in 0.32s, 4 tests Suite: org.elasticsearch.cluster.node.DiscoveryNodesTests Completed [518/1066] on J0 in 0.14s, 7 tests Suite: org.elasticsearch.index.mapper.KeywordFieldTypeTests Completed [519/1066] on J0 in 0.41s, 12 tests Suite: org.elasticsearch.index.mapper.CamelCaseFieldNameTests Completed [520/1066] on J1 in 2.83s, 1 test Suite: org.elasticsearch.search.aggregations.bucket.nested.ReverseNestedTests Completed [521/1066] on J0 in 0.50s, 5 tests Suite: org.elasticsearch.index.mapper.DocumentMapperMergeTests Completed [522/1066] on J2 in 3.99s, 8 tests Suite: org.elasticsearch.cluster.routing.allocation.SingleShardOneReplicaRoutingTests Completed [523/1066] on J1 in 0.17s, 1 test Suite: org.elasticsearch.search.aggregations.metrics.cardinality.CardinalityTests Completed [524/1066] on J2 in 0.06s, 5 tests Suite: org.elasticsearch.bootstrap.BootstrapTests Completed [525/1066] on J0 in 0.22s, 1 test Suite: org.elasticsearch.search.aggregations.metrics.geocentroid.GeoCentroidAggregatorTests Completed [526/1066] on J1 in 0.22s, 4 tests Suite: org.elasticsearch.common.xcontent.support.filtering.FilterPathTests Completed [527/1066] on J0 in 0.08s, 12 tests Suite: org.elasticsearch.index.engine.SegmentTests Completed [528/1066] on J0 in 0.05s, 1 test Suite: org.elasticsearch.common.xcontent.support.filtering.CborXContentFilteringTests Completed [529/1066] on J1 in 0.21s, 36 tests Suite: org.elasticsearch.common.unit.RatioValueTests Completed [530/1066] on J1 in 0.03s, 2 tests Suite: org.elasticsearch.index.fielddata.BinaryDVFieldDataTests Completed [531/1066] on J2 in 0.62s, 2 tests Suite: org.elasticsearch.common.xcontent.cbor.CborXContentParserTests Completed [532/1066] on J2 in 0.05s, 1 test Suite: org.elasticsearch.action.admin.indices.stats.IndicesStatsTests Completed [533/1066] on J0 in 1.21s, 4 tests Suite: org.elasticsearch.index.query.functionscore.FieldValueFactorFunctionModifierTests Completed [534/1066] on J0 in 0.04s, 4 tests Suite: org.elasticsearch.search.aggregations.pipeline.bucketsort.BucketSortTests Completed [535/1066] on J0 in 0.29s, 7 tests Suite: org.elasticsearch.index.mapper.RoutingFieldMapperTests Completed [536/1066] on J2 in 0.99s, 2 tests Suite: org.elasticsearch.index.shard.DocsStatsTests Completed [537/1066] on J2 in 0.02s, 3 tests Suite: org.elasticsearch.index.mapper.IndexFieldMapperTests Completed [538/1066] on J2 in 0.43s, 2 tests Suite: org.elasticsearch.index.get.DocumentFieldTests Completed [539/1066] on J2 in 0.04s, 3 tests Suite: org.elasticsearch.index.mapper.MapperTests Completed [540/1066] on J2 in 0.03s, 2 tests Suite: org.elasticsearch.index.mapper.FieldTypeLookupTests Completed [541/1066] on J2 in 0.04s, 10 tests Suite: org.elasticsearch.index.mapper.RangeFieldQueryStringQueryBuilderTests Completed [542/1066] on J2 in 0.32s, 15 tests Suite: org.elasticsearch.search.aggregations.support.MissingValuesTests Completed [543/1066] on J2 in 0.09s, 7 tests Suite: org.elasticsearch.repositories.blobstore.BlobStoreRepositoryTests Completed [544/1066] on J0 in 1.05s, 6 tests Suite: org.elasticsearch.index.mapper.TextFieldMapperTests Completed [545/1066] on J1 in 2.93s, 19 tests Suite: org.elasticsearch.index.fielddata.ordinals.SingleOrdinalsTests Completed [546/1066] on J2 in 0.53s, 2 tests Suite: org.elasticsearch.search.suggest.phrase.StupidBackoffModelTests Completed [547/1066] on J1 in 0.05s, 4 tests Suite: org.elasticsearch.index.mapper.DocumentFieldMapperTests Completed [548/1066] on J1 in 0.02s, 1 test Suite: org.elasticsearch.search.aggregations.metrics.avg.InternalAvgTests Completed [549/1066] on J1 in 0.14s, 6 tests Suite: org.elasticsearch.action.support.replication.TransportReplicationActionTests Completed [550/1066] on J0 in 1.07s, 19 tests Suite: org.elasticsearch.index.engine.VersionValueTests Completed [551/1066] on J1 in 0.21s, 2 tests Suite: org.elasticsearch.search.aggregations.metrics.StatsAggregatorTests Completed [552/1066] on J0 in 0.31s, 4 tests Suite: org.elasticsearch.repositories.blobstore.BlobStoreRepositoryRestoreTests Completed [553/1066] on J2 in 0.70s, 2 tests Suite: org.elasticsearch.common.xcontent.builder.XContentBuilderTests Completed [554/1066] on J1 in 0.15s, 21 tests Suite: org.elasticsearch.action.support.replication.ReplicationResponseTests Completed [555/1066] on J2 in 0.09s, 3 tests Suite: org.elasticsearch.cluster.serialization.ClusterStateToStringTests Completed [556/1066] on J2 in 0.11s, 1 test Suite: org.elasticsearch.client.transport.TransportClientTests Completed [557/1066] on J0 in 0.12s, 2 tests Suite: org.elasticsearch.common.network.NetworkAddressTests Completed [558/1066] on J0 in 0.13s, 7 tests Suite: org.elasticsearch.cluster.routing.operation.hash.murmur3.Murmur3HashFunctionTests Completed [559/1066] on J0 in 0.06s, 1 test Suite: org.elasticsearch.index.shard.PrimaryReplicaSyncerTests Completed [560/1066] on J2 in 0.32s, 5 tests Suite: org.elasticsearch.index.mapper.CompletionFieldTypeTests Completed [561/1066] on J0 in 0.07s, 5 tests Suite: org.elasticsearch.action.admin.indices.create.CreateIndexRequestBuilderTests Completed [562/1066] on J2 in 0.03s, 2 tests Suite: org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactoryTests Completed [564/1066] on J2 in 0.04s, 1 test Suite: org.elasticsearch.index.mapper.RootObjectMapperTests Completed [565/1066] on J1 in 1.07s, 4 tests Suite: org.elasticsearch.index.seqno.SequenceNumbersTests Completed [566/1066] on J1 in 0.03s, 2 tests Suite: org.elasticsearch.common.lucene.index.FreqTermsEnumTests Completed [567/1066] on J0 in 0.33s, 3 tests Suite: org.elasticsearch.search.aggregations.metrics.InternalMaxTests Completed [568/1066] on J0 in 0.08s, 5 tests Suite: org.elasticsearch.search.aggregations.bucket.FiltersTests Completed [569/1066] on J2 in 0.32s, 8 tests Suite: org.elasticsearch.action.get.MultiGetShardRequestTests Completed [570/1066] on J2 in 0.03s, 1 test Suite: org.elasticsearch.index.mapper.IgnoredFieldTypeTests Completed [571/1066] on J1 in 0.07s, 5 tests Suite: org.elasticsearch.search.SearchRequestTests Completed [572/1066] on J0 in 0.11s, 4 tests Suite: org.elasticsearch.cluster.service.MasterServiceTests Completed [573/1066] on J0 in 0.16s, 6 tests Suite: org.elasticsearch.index.query.MatchAllQueryBuilderTests Completed [574/1066] on J2 in 0.09s, 10 tests Suite: org.elasticsearch.bootstrap.BootstrapSettingsTests Completed [575/1066] on J2 in 0.02s, 1 test Suite: org.elasticsearch.search.aggregations.bucket.DateHistogramTests Completed [576/1066] on J2 in 0.13s, 5 tests Suite: org.elasticsearch.index.fieldstats.FieldStatsProviderRefreshTests Completed [577/1066] on J1 in 0.44s, 1 test Suite: org.elasticsearch.common.lucene.store.ByteArrayIndexInputTests Completed [578/1066] on J0 in 0.07s, 3 tests Suite: org.elasticsearch.action.admin.indices.create.CreateIndexRequestTests Completed [579/1066] on J0 in 0.04s, 4 tests Suite: org.elasticsearch.search.profile.query.CollectorResultTests Completed [580/1066] on J0 in 0.05s, 3 tests Suite: org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationActionTests Completed [581/1066] on J2 in 0.12s, 7 tests Suite: org.elasticsearch.search.sort.SortModeTests Completed [582/1066] on J0 in 0.03s, 2 tests Suite: org.elasticsearch.action.ingest.WritePipelineResponseTests Completed [583/1066] on J2 in 0.03s, 2 tests Suite: org.elasticsearch.cluster.routing.allocation.ExpectedShardSizeAllocationTests Completed [584/1066] on J2 in 0.03s, 2 tests Suite: org.elasticsearch.ingest.PipelineFactoryTests Completed [585/1066] on J2 in 0.06s, 10 tests Suite: org.elasticsearch.index.shard.ShardIdTests Completed [586/1066] on J0 in 0.02s, 2 tests Suite: org.elasticsearch.index.query.RangeQueryBuilderTests Completed [587/1066] on J2 in 0.28s, 25 tests Suite: org.elasticsearch.index.mapper.TypeFieldMapperTests Completed [588/1066] on J0 in 0.66s, 3 tests Suite: org.elasticsearch.search.aggregations.metrics.geobounds.InternalGeoBoundsTests Completed [589/1066] on J0 in 0.06s, 5 tests Suite: org.elasticsearch.common.bytes.CompositeBytesReferenceTests Completed [590/1066] on J0 in 0.53s, 31 tests Suite: org.elasticsearch.script.ScriptExceptionTests Completed [591/1066] on J0 in 0.02s, 4 tests Suite: org.elasticsearch.gateway.ReplicaShardAllocatorTests Completed [592/1066] on J0 in 0.07s, 15 tests Suite: org.elasticsearch.index.mapper.MapperServiceTests Completed [593/1066] on J1 in 1.50s, 14 tests Suite: org.elasticsearch.index.mapper.FieldNamesFieldTypeTests Completed [594/1066] on J1 in 0.03s, 6 tests Suite: org.elasticsearch.action.search.SearchRequestBuilderTests Completed [595/1066] on J0 in 0.05s, 4 tests Suite: org.elasticsearch.index.reindex.DeleteByQueryRequestTests Completed [596/1066] on J0 in 0.02s, 6 tests Suite: org.elasticsearch.search.aggregations.support.PathTests Completed [597/1066] on J0 in 0.02s, 2 tests Suite: org.elasticsearch.index.fielddata.FieldDataCacheTests Completed [598/1066] on J0 in 0.09s, 1 test Suite: org.elasticsearch.action.support.ListenableActionFutureTests Completed [599/1066] on J0 in 0.03s, 1 test Suite: org.elasticsearch.common.joda.JodaTests Completed [600/1066] on J0 in 0.03s, 1 test Suite: org.elasticsearch.common.recycler.LockedRecyclerTests Completed [601/1066] on J0 in 0.02s, 5 tests Suite: org.elasticsearch.search.suggest.phrase.LaplaceModelTests Completed [602/1066] on J0 in 0.03s, 4 tests Suite: org.elasticsearch.cluster.routing.UnassignedInfoTests Completed [603/1066] on J0 in 0.08s, 16 tests Suite: org.elasticsearch.client.transport.TransportClientNodesServiceTests Completed [604/1066] on J2 in 1.93s, 5 tests Suite: org.elasticsearch.search.aggregations.bucket.TermsTests Completed [605/1066] on J2 in 0.21s, 5 tests Suite: org.elasticsearch.rest.action.RestActionsTests Completed [606/1066] on J2 in 0.06s, 5 tests Suite: org.elasticsearch.cluster.routing.allocation.decider.DiskThresholdDeciderUnitTests Completed [607/1066] on J2 in 0.11s, 4 tests Suite: org.elasticsearch.index.query.MatchPhraseQueryBuilderTests Completed [608/1066] on J2 in 0.55s, 14 tests Suite: org.elasticsearch.index.mapper.NumberFieldTypeTests Completed [609/1066] on J0 in 1.55s, 32 tests Suite: org.elasticsearch.search.searchafter.SearchAfterBuilderTests Completed [610/1066] on J2 in 0.25s, 7 tests Suite: org.elasticsearch.index.mapper.RangeFieldMapperTests Completed [611/1066] on J1 in 2.85s, 11 tests Suite: org.elasticsearch.search.aggregations.support.IncludeExcludeTests Completed [612/1066] on J1 in 0.03s, 8 tests Suite: org.elasticsearch.index.fielddata.ordinals.MultiOrdinalsTests Completed [613/1066] on J1 in 0.22s, 3 tests Suite: org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregatorTests Completed [614/1066] on J0 in 1.71s, 12 tests Suite: org.elasticsearch.repositories.RepositoryDataTests Completed [615/1066] on J1 in 0.39s, 9 tests Suite: org.elasticsearch.search.profile.SearchProfileShardResultsTests Completed [616/1066] on J0 in 0.25s, 2 tests Suite: org.elasticsearch.common.settings.SecureStringTests Completed [617/1066] on J0 in 0.03s, 4 tests Suite: org.elasticsearch.action.support.replication.ReplicationOperationTests Completed [618/1066] on J1 in 0.39s, 5 tests Suite: org.elasticsearch.action.ActionListenerTests Completed [619/1066] on J0 in 0.03s, 3 tests Suite: org.elasticsearch.cluster.serialization.ClusterSerializationTests Completed [620/1066] on J1 in 0.10s, 4 tests Suite: org.elasticsearch.search.aggregations.metrics.percentiles.tdigest.InternalTDigestPercentilesTests Completed [621/1066] on J0 in 0.21s, 6 tests Suite: org.elasticsearch.index.shard.NewPathForShardTests Completed [622/1066] on J1 in 0.43s, 4 tests Suite: org.elasticsearch.index.query.NestedQueryBuilderTests Completed [623/1066] on J0 in 0.33s, 27 tests Suite: org.elasticsearch.action.admin.indices.close.CloseIndexResponseTests Completed [624/1066] on J1 in 0.11s, 3 tests Suite: org.elasticsearch.cluster.routing.allocation.BalancedSingleShardTests Completed [625/1066] on J0 in 0.11s, 8 tests Suite: org.elasticsearch.search.aggregations.bucket.composite.SingleDimensionValuesSourceTests Completed [626/1066] on J0 in 0.15s, 3 tests Suite: org.elasticsearch.search.aggregations.bucket.terms.StringTermsTests Completed [627/1066] on J1 in 0.20s, 6 tests Suite: org.elasticsearch.search.aggregations.bucket.FilterTests Completed [628/1066] on J1 in 0.04s, 5 tests Suite: org.elasticsearch.search.aggregations.metrics.percentiles.hdr.InternalHDRPercentilesRanksTests Completed [629/1066] on J0 in 0.59s, 6 tests Suite: org.elasticsearch.action.get.MultiGetResponseTests Completed [630/1066] on J1 in 0.36s, 1 test Suite: org.elasticsearch.index.query.InnerHitBuilderTests Completed [631/1066] on J1 in 0.64s, 4 tests Suite: org.elasticsearch.index.engine.LiveVersionMapTests Completed [632/1066] on J2 in 4.35s, 8 tests Suite: org.elasticsearch.bootstrap.BootstrapChecksTests Completed [633/1066] on J1 in 0.18s, 24 tests Suite: org.elasticsearch.index.mapper.DateFieldTypeTests Completed [634/1066] on J0 in 0.84s, 11 tests Suite: org.elasticsearch.search.aggregations.metrics.GeoBoundsTests Completed [635/1066] on J0 in 0.09s, 5 tests Suite: org.elasticsearch.search.aggregations.bucket.BucketUtilsTests Completed [636/1066] on J1 in 0.07s, 4 tests Suite: org.elasticsearch.action.support.replication.TransportWriteActionTests Completed [637/1066] on J1 in 0.11s, 9 tests Suite: org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeActionTests Completed [638/1066] on J0 in 0.18s, 6 tests Suite: org.elasticsearch.action.ingest.TrackingResultProcessorTests Completed [639/1066] on J1 in 0.06s, 4 tests Suite: org.elasticsearch.index.translog.TranslogHeaderTests Completed [640/1066] on J0 in 0.07s, 3 tests Suite: org.elasticsearch.cluster.routing.allocation.ElectReplicaAsPrimaryDuringRelocationTests Completed [641/1066] on J1 in 0.13s, 1 test Suite: org.elasticsearch.ingest.PipelineExecutionServiceTests Completed [642/1066] on J1 in 0.19s, 14 tests Suite: org.apache.lucene.analysis.miscellaneous.DeDuplicatingTokenFilterTests Completed [643/1066] on J0 in 0.21s, 3 tests Suite: org.elasticsearch.action.admin.indices.alias.IndicesAliasesRequestTests Completed [644/1066] on J0 in 0.11s, 1 test Suite: org.elasticsearch.index.query.QueryStringQueryBuilderTests Completed [645/1066] on J1 in 1.25s, 49 tests Suite: org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorTests Completed [646/1066] on J0 in 1.72s, 14 tests Suite: org.elasticsearch.client.transport.TransportClientHeadersTests Completed [647/1066] on J1 in 1.42s, 3 tests Suite: org.elasticsearch.search.aggregations.bucket.SignificantTextTests Completed [648/1066] on J1 in 0.11s, 5 tests Suite: org.elasticsearch.rest.RestUtilsTests Completed [649/1066] on J1 in 0.32s, 4 tests Suite: org.elasticsearch.cluster.routing.allocation.decider.DiskThresholdDeciderTests Completed [650/1066] on J1 in 0.24s, 9 tests Suite: org.elasticsearch.index.query.MatchPhrasePrefixQueryBuilderTests Completed [651/1066] on J1 in 0.16s, 13 tests Suite: org.elasticsearch.search.scroll.RestSearchScrollActionTests Completed [652/1066] on J1 in 0.12s, 2 tests Suite: org.elasticsearch.search.aggregations.bucket.missing.MissingAggregatorTests Completed [653/1066] on J0 in 3.35s, 4 tests Suite: org.elasticsearch.index.query.QueryShardContextTests Completed [654/1066] on J0 in 0.16s, 2 tests Suite: org.elasticsearch.rest.DeprecationRestHandlerTests Completed [655/1066] on J0 in 0.25s, 8 tests Suite: org.elasticsearch.search.aggregations.metrics.SumTests Completed [656/1066] on J0 in 0.07s, 5 tests Suite: org.elasticsearch.common.geo.builders.EnvelopeBuilderTests Completed [657/1066] on J0 in 0.05s, 4 tests Suite: org.elasticsearch.action.admin.indices.flush.FlushResponseTests Completed [658/1066] on J0 in 0.10s, 3 tests Suite: org.elasticsearch.common.xcontent.XContentTypeTests Completed [659/1066] on J0 in 0.34s, 8 tests Suite: org.elasticsearch.cli.MultiCommandTests Completed [660/1066] on J0 in 0.35s, 8 tests Suite: org.elasticsearch.index.fielddata.SortedSetDVStringFieldDataTests Completed [661/1066] on J2 in 7.60s, 19 tests Suite: org.elasticsearch.index.mapper.GeoPointFieldTypeTests Completed [662/1066] on J2 in 0.04s, 5 tests Suite: org.elasticsearch.index.query.TermsSetQueryBuilderTests Completed [663/1066] on J2 in 0.24s, 12 tests Suite: org.elasticsearch.search.aggregations.bucket.global.InternalGlobalTests Completed [664/1066] on J2 in 0.13s, 5 tests Suite: org.elasticsearch.index.mapper.SourceFieldTypeTests Completed [665/1066] on J2 in 0.03s, 5 tests Suite: org.elasticsearch.cluster.routing.OperationRoutingTests Completed [666/1066] on J2 in 0.25s, 10 tests Suite: org.elasticsearch.index.search.geo.GeoPointParsingTests Completed [667/1066] on J2 in 0.05s, 7 tests Suite: org.elasticsearch.search.aggregations.bucket.filter.FiltersAggregatorTests Completed [668/1066] on J2 in 1.40s, 4 tests Suite: org.elasticsearch.discovery.zen.PublishClusterStateActionTests Completed [669/1066] on J0 in 8.10s, 10 tests Suite: org.elasticsearch.index.fielddata.FilterFieldDataTests IGNOR/A 0.11s J0 | FilterFieldDataTests.testEmpty > Assumption #1: No need to test empty usage here Completed [670/1066] on J0 in 0.45s, 2 tests, 1 skipped Suite: org.elasticsearch.ExceptionSerializationTests Completed [671/1066] on J2 in 6.13s, 41 tests Suite: org.elasticsearch.common.xcontent.support.filtering.YamlFilteringGeneratorTests Completed [672/1066] on J0 in 0.25s, 36 tests Suite: org.elasticsearch.search.aggregations.pipeline.moving.avg.MovAvgUnitTests Completed [673/1066] on J0 in 0.08s, 13 tests Suite: org.elasticsearch.index.replication.IndexLevelReplicationTests Completed [674/1066] on J2 in 5.02s, 11 tests Suite: org.elasticsearch.search.aggregations.pipeline.bucketmetrics.StatsBucketTests Completed [675/1066] on J2 in 0.06s, 4 tests Suite: org.elasticsearch.gateway.PriorityComparatorTests Completed [676/1066] on J2 in 0.06s, 3 tests Suite: org.elasticsearch.monitor.process.ProcessProbeTests Completed [677/1066] on J2 in 0.07s, 2 tests Suite: org.elasticsearch.index.fielddata.IndexFieldDataServiceTests Completed [678/1066] on J2 in 0.55s, 6 tests Suite: org.elasticsearch.search.aggregations.metrics.tophits.InternalTopHitsTests Completed [679/1066] on J2 in 0.09s, 5 tests Suite: org.elasticsearch.index.query.PrefixQueryBuilderTests Completed [680/1066] on J2 in 0.09s, 14 tests Suite: org.elasticsearch.index.analysis.AnalysisRegistryTests Completed [681/1066] on J2 in 0.25s, 8 tests Suite: org.elasticsearch.rest.BytesRestResponseTests Completed [682/1066] on J2 in 0.07s, 12 tests Suite: org.elasticsearch.search.aggregations.metrics.AvgTests Completed [683/1066] on J2 in 0.07s, 5 tests Suite: org.elasticsearch.search.DocValueFormatTests Completed [684/1066] on J2 in 0.19s, 9 tests Suite: org.elasticsearch.search.internal.AliasFilterTests Completed [685/1066] on J2 in 0.03s, 1 test Suite: org.elasticsearch.search.aggregations.support.ValueTypeTests Completed [686/1066] on J2 in 0.03s, 2 tests Suite: org.elasticsearch.common.settings.SettingsFilterTests Completed [687/1066] on J2 in 0.05s, 4 tests Suite: org.elasticsearch.indices.recovery.RecoveryTests Completed [688/1066] on J0 in 6.53s, 8 tests Suite: org.elasticsearch.cluster.routing.allocation.ConcurrentRebalanceRoutingTests Completed [689/1066] on J2 in 0.10s, 1 test Suite: org.elasticsearch.action.ingest.SimulatePipelineResponseTests Completed [690/1066] on J2 in 0.03s, 1 test Suite: org.elasticsearch.search.aggregations.bucket.significant.SignificantStringTermsTests Completed [691/1066] on J2 in 0.12s, 6 tests Suite: org.elasticsearch.search.aggregations.bucket.significant.SignificanceHeuristicTests Completed [692/1066] on J0 in 0.06s, 7 tests Suite: org.elasticsearch.snapshots.SnapshotTests Completed [693/1066] on J2 in 0.11s, 4 tests Suite: org.elasticsearch.action.support.IndicesOptionsTests Completed [694/1066] on J2 in 0.06s, 4 tests Suite: org.elasticsearch.index.mapper.PathMapperTests Completed [695/1066] on J2 in 0.40s, 1 test Suite: org.elasticsearch.discovery.zen.NodeJoinControllerTests Completed [696/1066] on J0 in 0.67s, 14 tests Suite: org.elasticsearch.action.admin.indices.shrink.ResizeResponseTests Completed [697/1066] on J0 in 0.17s, 4 tests Suite: org.elasticsearch.index.query.TermQueryBuilderTests IGNOR/A 0.00s J0 | TermQueryBuilderTests.testGeo > Assumption #1: test runs only when at least a type is registered Completed [698/1066] on J0 in 0.12s, 14 tests, 1 skipped Suite: org.elasticsearch.index.cache.bitset.BitSetFilterCacheTests Completed [699/1066] on J0 in 0.09s, 4 tests Suite: org.elasticsearch.action.support.DefaultShardOperationFailedExceptionTests Completed [700/1066] on J0 in 0.03s, 3 tests Suite: org.elasticsearch.index.query.IdsQueryBuilderTests Completed [701/1066] on J0 in 0.12s, 12 tests Suite: org.elasticsearch.index.IndexingSlowLogTests Completed [702/1066] on J0 in 0.06s, 4 tests Suite: org.elasticsearch.search.sort.ScoreSortBuilderTests Completed [703/1066] on J0 in 0.03s, 7 tests Suite: org.elasticsearch.common.PidFileTests Completed [704/1066] on J0 in 0.03s, 2 tests Suite: org.elasticsearch.action.bulk.BulkProcessorTests Completed [705/1066] on J2 in 1.11s, 2 tests Suite: org.elasticsearch.common.collect.EvictingQueueTests Completed [706/1066] on J2 in 0.07s, 8 tests Suite: org.elasticsearch.common.xcontent.support.XContentHelperTests Completed [707/1066] on J2 in 0.08s, 2 tests Suite: org.elasticsearch.persistent.decider.EnableAssignmentDeciderTests Completed [708/1066] on J2 in 0.06s, 2 tests Suite: org.elasticsearch.common.util.IndexFolderUpgraderTests Completed [709/1066] on J0 in 0.61s, 5 tests Suite: org.elasticsearch.threadpool.FixedThreadPoolTests Completed [710/1066] on J0 in 0.04s, 1 test Suite: org.elasticsearch.bootstrap.MaxMapCountCheckTests Completed [711/1066] on J0 in 0.06s, 4 tests Suite: org.elasticsearch.common.xcontent.UnknownNamedObjectExceptionTests Completed [712/1066] on J0 in 0.06s, 2 tests Suite: org.elasticsearch.cluster.routing.allocation.TenShardsOneReplicaRoutingTests Completed [713/1066] on J0 in 0.09s, 1 test Suite: org.elasticsearch.cluster.routing.allocation.AllocationCommandsTests Completed [714/1066] on J0 in 0.17s, 7 tests Suite: org.elasticsearch.cluster.routing.allocation.PrimaryNotRelocatedWhileBeingRecoveredTests Completed [715/1066] on J0 in 0.03s, 1 test Suite: org.elasticsearch.index.query.MatchQueryBuilderTests Completed [716/1066] on J0 in 0.29s, 18 tests Suite: org.elasticsearch.index.mapper.IpRangeFieldMapperTests Completed [717/1066] on J0 in 0.72s, 1 test Suite: org.elasticsearch.index.shard.IndexSearcherWrapperTests Completed [718/1066] on J0 in 0.09s, 3 tests Suite: org.elasticsearch.common.lucene.uid.VersionsTests Completed [719/1066] on J0 in 0.09s, 4 tests Suite: org.elasticsearch.BuildTests Completed [720/1066] on J0 in 0.06s, 2 tests Suite: org.elasticsearch.search.fetch.subphase.FetchSourceSubPhaseTests Completed [721/1066] on J0 in 0.04s, 4 tests Suite: org.elasticsearch.common.regex.RegexTests Completed [722/1066] on J0 in 0.03s, 2 tests Suite: org.elasticsearch.index.mapper.DateFieldMapperTests Completed [723/1066] on J2 in 2.68s, 14 tests Suite: org.apache.lucene.queries.IntegerRandomBinaryDocValuesRangeQueryTests IGNOR/A 0.00s J2 | IntegerRandomBinaryDocValuesRangeQueryTests.testRandomBig > Assumption #1: 'nightly' test group is disabled (@Nightly()) Completed [724/1066] on J2 in 1.89s, 4 tests, 1 skipped Suite: org.elasticsearch.search.suggest.completion.CategoryContextMappingTests Completed [725/1066] on J0 in 2.77s, 27 tests Suite: org.elasticsearch.index.analysis.AnalysisTests Completed [726/1066] on J0 in 0.05s, 4 tests Suite: org.elasticsearch.client.node.NodeClientHeadersTests Completed [727/1066] on J0 in 0.04s, 2 tests Suite: org.elasticsearch.monitor.jvm.JvmGcMonitorServiceTests Completed [728/1066] on J2 in 0.08s, 2 tests Suite: org.elasticsearch.common.util.concurrent.TimedRunnableTests Completed [729/1066] on J0 in 0.05s, 2 tests Suite: org.elasticsearch.cluster.routing.allocation.PrimaryElectionRoutingTests Completed [730/1066] on J2 in 0.06s, 2 tests Suite: org.elasticsearch.action.admin.cluster.node.stats.NodeStatsTests Completed [731/1066] on J2 in 0.04s, 1 test Suite: org.elasticsearch.index.mapper.IdFieldTypeTests Completed [732/1066] on J0 in 0.41s, 7 tests Suite: org.elasticsearch.search.aggregations.metrics.CardinalityAggregatorTests Completed [733/1066] on J0 in 0.53s, 6 tests Suite: org.elasticsearch.index.mapper.MultiFieldCopyToMapperTests Completed [734/1066] on J0 in 0.03s, 1 test Suite: org.elasticsearch.index.engine.CombinedDeletionPolicyTests Completed [735/1066] on J2 in 0.99s, 5 tests Suite: org.elasticsearch.action.ingest.PutPipelineRequestTests Completed [736/1066] on J0 in 0.11s, 1 test Suite: org.apache.lucene.queries.BlendedTermQueryTests Completed [737/1066] on J2 in 0.20s, 3 tests Suite: org.elasticsearch.index.engine.InternalEngineTests Completed [738/1066] on J1 in 26.64s, 85 tests Suite: org.elasticsearch.index.mapper.PathMatchDynamicTemplateTests Completed [739/1066] on J0 in 0.51s, 1 test Suite: org.elasticsearch.search.internal.ScrollContextTests Completed [740/1066] on J0 in 0.26s, 1 test Suite: org.apache.lucene.queries.DoubleRandomBinaryDocValuesRangeQueryTests IGNOR/A 0.00s J2 | DoubleRandomBinaryDocValuesRangeQueryTests.testRandomBig > Assumption #1: 'nightly' test group is disabled (@Nightly()) Completed [741/1066] on J2 in 1.85s, 4 tests, 1 skipped Suite: org.apache.lucene.queries.MinDocQueryTests Completed [742/1066] on J2 in 0.20s, 3 tests Suite: org.elasticsearch.index.query.RangeQueryRewriteTests Completed [743/1066] on J0 in 1.76s, 3 tests Suite: org.elasticsearch.action.admin.indices.rollover.RolloverRequestTests Completed [744/1066] on J0 in 0.05s, 7 tests Suite: org.elasticsearch.search.aggregations.bucket.GeoDistanceRangeTests Completed [745/1066] on J0 in 0.19s, 7 tests Suite: org.apache.lucene.queries.SearchAfterSortedDocQueryTests Completed [746/1066] on J2 in 0.28s, 3 tests Suite: org.apache.lucene.search.uhighlight.CustomPassageFormatterTests Completed [747/1066] on J2 in 0.04s, 2 tests Suite: org.apache.lucene.search.uhighlight.CustomUnifiedHighlighterTests Completed [748/1066] on J2 in 0.36s, 9 tests Suite: org.elasticsearch.action.ShardValidateQueryRequestTests Completed [749/1066] on J2 in 0.02s, 1 test Suite: org.elasticsearch.action.admin.cluster.allocation.ClusterAllocationExplainActionTests Completed [750/1066] on J2 in 0.03s, 5 tests Suite: org.elasticsearch.index.mapper.DynamicMappingTests Completed [751/1066] on J0 in 1.55s, 22 tests Suite: org.elasticsearch.action.bulk.TransportBulkActionIngestTests Completed [752/1066] on J0 in 0.20s, 6 tests Suite: org.elasticsearch.action.admin.cluster.allocation.ClusterAllocationExplainRequestTests Completed [753/1066] on J2 in 0.04s, 1 test Suite: org.elasticsearch.rest.RestRequestTests Completed [754/1066] on J0 in 0.05s, 12 tests Suite: org.elasticsearch.action.admin.cluster.health.ClusterHealthResponsesTests Completed [755/1066] on J2 in 0.02s, 2 tests Suite: org.elasticsearch.action.admin.cluster.node.tasks.TaskTests Completed [756/1066] on J2 in 0.02s, 1 test Suite: org.elasticsearch.action.bulk.TransportShardBulkActionTests Completed [757/1066] on J0 in 0.94s, 16 tests Suite: org.elasticsearch.index.query.GeoDistanceQueryBuilderTests Completed [758/1066] on J0 in 0.15s, 25 tests Suite: org.elasticsearch.monitor.jvm.JvmInfoTests Completed [759/1066] on J0 in 0.02s, 1 test Suite: org.elasticsearch.plugins.PluginsTests Completed [760/1066] on J0 in 0.03s, 2 tests Suite: org.elasticsearch.search.aggregations.bucket.sampler.DiversifiedSamplerTests Completed [761/1066] on J0 in 0.14s, 2 tests Suite: org.elasticsearch.cluster.health.ClusterIndexHealthTests Completed [762/1066] on J0 in 0.07s, 1 test Suite: org.elasticsearch.cluster.GetClusterStateTests Completed [763/1066] on J0 in 0.23s, 2 tests Suite: org.elasticsearch.index.mapper.BinaryFieldTypeTests Completed [764/1066] on J0 in 0.03s, 5 tests Suite: org.elasticsearch.common.util.concurrent.KeyedLockTests Completed [765/1066] on J0 in 0.12s, 4 tests Suite: org.elasticsearch.common.util.concurrent.ResizableBlockingQueueTests Completed [766/1066] on J0 in 0.03s, 1 test Suite: org.elasticsearch.cluster.routing.allocation.SameShardRoutingTests Completed [767/1066] on J0 in 0.03s, 2 tests Suite: org.elasticsearch.index.mapper.RangeFieldTypeTests Completed [768/1066] on J0 in 0.17s, 8 tests Suite: org.elasticsearch.persistent.PersistentTasksCustomMetaDataTests Completed [769/1066] on J0 in 0.17s, 6 tests Suite: org.elasticsearch.index.get.GetResultTests Completed [770/1066] on J0 in 0.19s, 7 tests Suite: org.elasticsearch.index.query.FieldMaskingSpanQueryBuilderTests Completed [771/1066] on J0 in 0.10s, 11 tests Suite: org.elasticsearch.index.shard.IndexingOperationListenerTests Completed [772/1066] on J0 in 0.10s, 1 test Suite: org.elasticsearch.index.query.SpanMultiTermQueryBuilderTests Completed [773/1066] on J0 in 0.26s, 13 tests Suite: org.elasticsearch.action.admin.cluster.reroute.ClusterRerouteRequestTests Completed [774/1066] on J0 in 0.09s, 3 tests Suite: org.elasticsearch.action.admin.cluster.state.ClusterStateRequestTests Completed [775/1066] on J0 in 0.02s, 1 test Suite: org.elasticsearch.action.admin.indices.alias.IndicesAliasesResponseTests Completed [776/1066] on J0 in 0.03s, 3 tests Suite: org.elasticsearch.action.admin.indices.analyze.AnalyzeRequestTests Completed [777/1066] on J0 in 0.02s, 2 tests Suite: org.elasticsearch.action.admin.indices.cache.clear.ClearIndicesCacheResponseTests Completed [778/1066] on J0 in 0.23s, 3 tests Suite: org.elasticsearch.action.admin.indices.create.CreateIndexResponseTests Completed [779/1066] on J0 in 0.06s, 6 tests Suite: org.elasticsearch.action.admin.indices.delete.DeleteIndexResponseTests Completed [780/1066] on J0 in 0.04s, 4 tests Suite: org.elasticsearch.action.admin.cluster.node.tasks.TransportTasksActionTests Completed [781/1066] on J2 in 5.86s, 9 tests Suite: org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponseTests Completed [782/1066] on J2 in 0.62s, 5 tests HEARTBEAT J1 PID(6380@SLAVE-0633DD87F): 2018-05-05T08:30:16, stalled for 11.8s at: ScriptDocValuesLongsTests.testDates Suite: org.elasticsearch.action.admin.indices.settings.put.UpdateSettingsRequestStreamableTests Completed [783/1066] on J2 in 0.13s, 2 tests Suite: org.elasticsearch.action.admin.indices.settings.put.UpdateSettingsRequestTests Completed [784/1066] on J2 in 0.10s, 1 test Suite: org.elasticsearch.action.admin.indices.settings.put.UpdateSettingsResponseTests Completed [785/1066] on J2 in 0.07s, 3 tests Suite: org.elasticsearch.action.admin.indices.segments.IndicesSegmentsRequestTests Completed [786/1066] on J0 in 3.60s, 5 tests Suite: org.elasticsearch.action.bulk.BulkRequestModifierTests Completed [787/1066] on J2 in 0.04s, 3 tests Suite: org.elasticsearch.action.bulk.RetryTests Completed [788/1066] on J0 in 0.07s, 4 tests Suite: org.elasticsearch.action.bulk.TransportBulkActionIndicesThatCannotBeCreatedTests Completed [789/1066] on J2 in 0.04s, 3 tests Suite: org.elasticsearch.action.fieldcaps.FieldCapabilitiesResponseTests Completed [790/1066] on J0 in 0.09s, 4 tests Suite: org.elasticsearch.action.get.MultiGetRequestTests Completed [791/1066] on J2 in 0.18s, 5 tests Suite: org.elasticsearch.action.index.IndexRequestBuilderTests Completed [792/1066] on J0 in 0.04s, 1 test Suite: org.elasticsearch.action.ingest.SimulateProcessorResultTests Completed [793/1066] on J2 in 0.02s, 1 test Suite: org.elasticsearch.action.ingest.WriteableIngestDocumentTests Completed [794/1066] on J0 in 0.02s, 3 tests Suite: org.elasticsearch.action.resync.ResyncReplicationRequestTests Completed [795/1066] on J2 in 0.02s, 1 test Suite: org.elasticsearch.action.search.SearchAsyncActionTests Completed [796/1066] on J0 in 0.04s, 3 tests Suite: org.elasticsearch.action.search.ShardSearchFailureTests Completed [797/1066] on J2 in 0.02s, 3 tests Suite: org.elasticsearch.action.search.TransportMultiSearchActionTests Completed [798/1066] on J0 in 0.07s, 2 tests Suite: org.elasticsearch.action.search.TransportSearchActionTests Completed [799/1066] on J2 in 0.12s, 3 tests Suite: org.elasticsearch.action.support.AdapterActionFutureTests Completed [800/1066] on J0 in 0.01s, 1 test Suite: org.elasticsearch.action.support.AutoCreateIndexTests Completed [801/1066] on J2 in 0.04s, 14 tests Suite: org.elasticsearch.action.support.ContextPreservingActionListenerTests Completed [802/1066] on J0 in 0.02s, 3 tests Suite: org.elasticsearch.action.support.RefreshPolicyTests Completed [803/1066] on J2 in 0.02s, 4 tests Suite: org.elasticsearch.action.support.TransportActionFilterChainTests Completed [804/1066] on J0 in 0.04s, 2 tests Suite: org.elasticsearch.action.support.replication.BroadcastReplicationTests Completed [805/1066] on J2 in 0.23s, 5 tests Suite: org.elasticsearch.cluster.ClusterChangedEventTests Completed [806/1066] on J0 in 0.05s, 8 tests Suite: org.elasticsearch.cluster.ClusterInfoTests Completed [807/1066] on J2 in 0.02s, 1 test Suite: org.elasticsearch.cluster.ClusterStateTests Completed [808/1066] on J0 in 0.02s, 1 test Suite: org.elasticsearch.cluster.DiskUsageTests Completed [809/1066] on J2 in 0.03s, 5 tests Suite: org.elasticsearch.cluster.action.shard.ShardStateActionTests Completed [810/1066] on J0 in 0.20s, 7 tests Suite: org.elasticsearch.cluster.block.ClusterBlockTests Completed [811/1066] on J2 in 0.02s, 3 tests Suite: org.elasticsearch.index.fielddata.ScriptDocValuesLongsTests Completed [812/1066] on J1 in 16.31s, 2 tests Suite: org.elasticsearch.cluster.health.ClusterStateHealthTests Completed [813/1066] on J0 in 0.05s, 8 tests Suite: org.elasticsearch.cluster.metadata.AliasMetaDataTests Completed [814/1066] on J2 in 0.02s, 1 test Suite: org.elasticsearch.cluster.metadata.IndexGraveyardTests Completed [815/1066] on J0 in 0.03s, 7 tests Suite: org.elasticsearch.cluster.metadata.IndexTemplateMetaDataTests Completed [816/1066] on J2 in 0.02s, 3 tests Suite: org.elasticsearch.cluster.metadata.MetaDataCreateIndexServiceTests Completed [817/1066] on J0 in 0.10s, 8 tests Suite: org.elasticsearch.cluster.metadata.MetaDataDeleteIndexServiceTests Completed [818/1066] on J2 in 0.04s, 3 tests Suite: org.elasticsearch.cluster.metadata.MetaDataIndexAliasesServiceTests Completed [819/1066] on J0 in 0.04s, 4 tests Suite: org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeServiceTests Completed [820/1066] on J2 in 0.04s, 6 tests Suite: org.elasticsearch.cluster.metadata.WildcardExpressionResolverTests Completed [821/1066] on J2 in 0.14s, 7 tests Suite: org.elasticsearch.cluster.node.DiscoveryNodeFiltersTests Completed [822/1066] on J2 in 0.09s, 16 tests Suite: org.elasticsearch.cluster.metadata.MetaDataMappingServiceTests Completed [823/1066] on J0 in 0.55s, 2 tests Suite: org.elasticsearch.cluster.node.DiscoveryNodeTests Completed [824/1066] on J2 in 0.02s, 2 tests Suite: org.elasticsearch.cluster.routing.allocation.AddIncrementallyTests Completed [825/1066] on J0 in 0.16s, 3 tests Suite: org.elasticsearch.cluster.routing.allocation.AllocationDecisionTests Completed [826/1066] on J0 in 0.09s, 4 tests Suite: org.elasticsearch.cluster.routing.allocation.AllocateUnassignedDecisionTests Completed [827/1066] on J2 in 0.61s, 6 tests Suite: org.elasticsearch.cluster.routing.allocation.AllocationPriorityTests Completed [828/1066] on J0 in 0.03s, 1 test Suite: org.elasticsearch.cluster.routing.allocation.BalanceConfigurationTests Completed [829/1066] on J2 in 0.33s, 4 tests Suite: org.elasticsearch.cluster.routing.allocation.InSyncAllocationIdTests Completed [830/1066] on J2 in 0.98s, 6 tests Suite: org.elasticsearch.index.mapper.GeoPointFieldMapperTests Completed [831/1066] on J1 in 4.16s, 16 tests Suite: org.elasticsearch.cluster.routing.allocation.BalanceUnbalancedClusterTests Completed [832/1066] on J0 in 1.63s, 1 test Suite: org.elasticsearch.cluster.routing.allocation.IndexBalanceTests Completed [833/1066] on J2 in 0.39s, 3 tests Suite: org.elasticsearch.index.query.TermsQueryBuilderTests IGNOR/A 0.00s J1 | TermsQueryBuilderTests.testGeo > Assumption #1: test runs only when at least a type is registered Completed [834/1066] on J1 in 0.40s, 19 tests, 1 skipped Suite: org.elasticsearch.search.aggregations.bucket.geogrid.InternalGeoHashGridTests Completed [835/1066] on J1 in 0.05s, 6 tests Suite: org.elasticsearch.cluster.routing.allocation.MaxRetryAllocationDeciderTests Completed [836/1066] on J0 in 0.15s, 2 tests Suite: org.elasticsearch.cluster.routing.allocation.NodeAllocationResultTests Completed [837/1066] on J2 in 0.04s, 2 tests Suite: org.elasticsearch.cluster.routing.allocation.ReplicaAllocatedAfterPrimaryTests Completed [838/1066] on J2 in 0.07s, 1 test Suite: org.elasticsearch.cluster.routing.allocation.NodeVersionAllocationDeciderTests Completed [839/1066] on J0 in 0.53s, 6 tests Suite: org.elasticsearch.cluster.routing.allocation.SingleShardNoReplicasRoutingTests Completed [840/1066] on J2 in 0.39s, 4 tests Suite: org.elasticsearch.cluster.serialization.DiffableTests Completed [841/1066] on J0 in 0.08s, 3 tests Suite: org.elasticsearch.cluster.structure.RoutingIteratorTests Completed [842/1066] on J2 in 0.09s, 8 tests Suite: org.elasticsearch.index.mapper.SourceFieldMapperTests Completed [843/1066] on J1 in 1.37s, 8 tests Suite: org.elasticsearch.cluster.routing.IndexShardRoutingTableTests Completed [844/1066] on J1 in 0.02s, 2 tests Suite: org.elasticsearch.index.search.SimpleQueryStringQueryParserTests Completed [845/1066] on J1 in 0.02s, 1 test Suite: org.elasticsearch.common.StringsTests Completed [846/1066] on J0 in 0.02s, 4 tests Suite: org.elasticsearch.common.blobstore.BlobPathTests Completed [847/1066] on J2 in 0.01s, 1 test Suite: org.elasticsearch.search.aggregations.bucket.filter.FilterAggregatorTests Completed [848/1066] on J1 in 0.45s, 3 tests Suite: org.elasticsearch.common.blobstore.FsBlobStoreContainerTests Completed [849/1066] on J0 in 0.11s, 4 tests Suite: org.elasticsearch.common.blobstore.FsBlobStoreTests Completed [850/1066] on J2 in 0.05s, 2 tests Suite: org.elasticsearch.ElasticsearchExceptionTests Completed [851/1066] on J1 in 0.14s, 16 tests Suite: org.elasticsearch.common.breaker.MemoryCircuitBreakerTests Completed [852/1066] on J0 in 0.22s, 5 tests Suite: org.elasticsearch.common.bytes.ByteBufferReferenceTests Completed [853/1066] on J2 in 0.50s, 30 tests Suite: org.elasticsearch.index.reindex.WorkerBulkByScrollTaskStateTests Completed [854/1066] on J1 in 0.54s, 5 tests Suite: org.elasticsearch.search.aggregations.pipeline.bucketmetrics.PercentilesBucketTests Completed [855/1066] on J1 in 0.38s, 5 tests Suite: org.elasticsearch.gateway.PrimaryShardAllocatorTests Completed [856/1066] on J1 in 0.10s, 18 tests Suite: org.elasticsearch.monitor.os.OsStatsTests Completed [857/1066] on J1 in 0.04s, 1 test Suite: org.elasticsearch.common.bytes.BytesArrayTests Completed [858/1066] on J0 in 0.72s, 31 tests Suite: org.elasticsearch.common.bytes.PagedBytesReferenceTests Completed [859/1066] on J2 in 0.68s, 32 tests Suite: org.elasticsearch.common.cache.CacheBuilderTests Completed [860/1066] on J0 in 0.02s, 2 tests Suite: org.elasticsearch.index.fielddata.GeoFieldDataTests IGNOR/A 0.06s J1 | GeoFieldDataTests.testSortMultiValuesFields > Assumption #1: Only test on non geo_point fields Completed [861/1066] on J1 in 0.62s, 8 tests, 1 skipped Suite: org.elasticsearch.search.aggregations.metrics.scripted.ScriptedMetricAggregatorTests Completed [862/1066] on J1 in 0.31s, 6 tests Suite: org.elasticsearch.common.geo.GeoJsonShapeParserTests Completed [863/1066] on J0 in 0.53s, 23 tests Suite: org.elasticsearch.index.query.OperatorTests Completed [864/1066] on J1 in 0.03s, 6 tests Suite: org.elasticsearch.index.VersionTypeTests Completed [865/1066] on J1 in 0.05s, 5 tests Suite: org.elasticsearch.rest.BaseRestHandlerTests Completed [866/1066] on J1 in 0.04s, 6 tests Suite: org.elasticsearch.search.aggregations.metrics.AdjacencyMatrixTests Completed [867/1066] on J1 in 0.06s, 6 tests Suite: org.elasticsearch.common.geo.GeoHashTests Completed [868/1066] on J2 in 1.00s, 4 tests Suite: org.elasticsearch.common.geo.SpatialStrategyTests Completed [869/1066] on J0 in 0.03s, 4 tests Suite: org.elasticsearch.search.DefaultSearchContextTests Completed [870/1066] on J1 in 0.08s, 1 test Suite: org.elasticsearch.common.geo.builders.CircleBuilderTests Completed [871/1066] on J2 in 0.05s, 3 tests Suite: org.elasticsearch.common.geo.builders.LineStringBuilderTests Completed [872/1066] on J2 in 0.10s, 4 tests Suite: org.elasticsearch.common.geo.builders.GeometryCollectionBuilderTests Completed [873/1066] on J0 in 0.26s, 3 tests Suite: org.elasticsearch.common.geo.builders.PolygonBuilderTests Completed [874/1066] on J2 in 0.07s, 7 tests Suite: org.elasticsearch.common.hppc.HppcMapsTests Completed [875/1066] on J2 in 0.02s, 1 test Suite: org.elasticsearch.common.hash.MessageDigestsTests Completed [876/1066] on J0 in 0.38s, 4 tests Suite: org.elasticsearch.common.io.FileSystemUtilsTests Completed [877/1066] on J2 in 0.07s, 4 tests Suite: org.elasticsearch.common.io.stream.ReleasableBytesStreamOutputTests Completed [878/1066] on J0 in 0.04s, 1 test Suite: org.elasticsearch.common.lucene.store.InputStreamIndexInputTests Completed [879/1066] on J0 in 0.17s, 6 tests Suite: org.elasticsearch.common.lucene.uid.VersionLookupTests Completed [880/1066] on J0 in 0.03s, 2 tests Suite: org.elasticsearch.common.lucene.LuceneTests Completed [881/1066] on J2 in 1.17s, 8 tests Suite: org.elasticsearch.search.geo.GeoShapeQueryTests Completed [882/1066] on J1 in 2.64s, 12 tests Suite: org.elasticsearch.search.aggregations.support.ScriptValuesTests Completed [883/1066] on J1 in 0.04s, 4 tests Suite: org.elasticsearch.common.settings.SettingTests Completed [884/1066] on J1 in 0.12s, 34 tests Suite: org.elasticsearch.cluster.routing.allocation.ClusterRebalanceRoutingTests Completed [885/1066] on J1 in 0.14s, 8 tests Suite: org.elasticsearch.action.ingest.SimulatePipelineRequestTests Completed [886/1066] on J1 in 0.02s, 3 tests Suite: org.elasticsearch.search.aggregations.bucket.significant.SignificantLongTermsTests Completed [887/1066] on J1 in 0.23s, 6 tests Suite: org.elasticsearch.snapshots.SnapshotRequestsTests Completed [888/1066] on J1 in 0.04s, 2 tests Suite: org.elasticsearch.action.support.GroupedActionListenerTests Completed [889/1066] on J1 in 0.09s, 2 tests Suite: org.elasticsearch.index.mapper.ObjectMapperTests Completed [890/1066] on J1 in 1.37s, 9 tests Suite: org.elasticsearch.common.settings.AddFileKeyStoreCommandTests Completed [891/1066] on J0 in 2.58s, 13 tests Suite: org.elasticsearch.action.bulk.BulkItemResponseTests Completed [892/1066] on J1 in 0.04s, 3 tests Suite: org.elasticsearch.common.collect.CopyOnWriteHashMapTests Completed [893/1066] on J1 in 0.09s, 4 tests Suite: org.elasticsearch.common.settings.CreateKeyStoreCommandTests Completed [894/1066] on J0 in 0.34s, 3 tests Suite: org.elasticsearch.common.xcontent.smile.SmileXContentTests Completed [895/1066] on J1 in 0.32s, 50 tests Suite: org.elasticsearch.persistent.decider.AssignmentDecisionTests Completed [896/1066] on J1 in 0.03s, 2 tests Suite: org.elasticsearch.common.unit.ByteSizeUnitTests Completed [897/1066] on J0 in 0.03s, 8 tests Suite: org.elasticsearch.common.unit.ByteSizeValueTests Completed [898/1066] on J0 in 0.21s, 27 tests Suite: org.elasticsearch.common.unit.DistanceUnitTests Completed [899/1066] on J0 in 0.05s, 5 tests Suite: org.elasticsearch.common.settings.AddStringKeyStoreCommandTests Completed [900/1066] on J2 in 3.51s, 13 tests Suite: org.elasticsearch.common.util.CancellableThreadsTests Completed [901/1066] on J2 in 0.33s, 1 test Suite: org.elasticsearch.index.mapper.CopyToMapperTests Completed [902/1066] on J1 in 1.59s, 16 tests Suite: org.apache.lucene.queries.InetAddressRandomBinaryDocValuesRangeQueryTests IGNOR/A 0.00s J1 | InetAddressRandomBinaryDocValuesRangeQueryTests.testRandomBig > Assumption #1: 'nightly' test group is disabled (@Nightly()) Completed [903/1066] on J1 in 0.81s, 4 tests, 1 skipped Suite: org.elasticsearch.monitor.jvm.JvmGcMonitorServiceSettingsTests Completed [904/1066] on J1 in 0.30s, 5 tests Suite: org.elasticsearch.cluster.routing.allocation.PreferPrimaryAllocationTests Completed [905/1066] on J1 in 0.04s, 1 test Suite: org.elasticsearch.action.admin.cluster.health.TransportClusterHealthActionTests Completed [906/1066] on J1 in 0.02s, 1 test Suite: org.elasticsearch.common.util.CollectionUtilsTests Completed [907/1066] on J2 in 0.81s, 10 tests Suite: org.elasticsearch.index.codec.CodecTests Completed [908/1066] on J1 in 0.03s, 3 tests Suite: org.elasticsearch.common.util.BytesRefHashTests Completed [909/1066] on J0 in 2.11s, 5 tests Suite: org.elasticsearch.common.util.concurrent.AbstractLifecycleRunnableTests Completed [910/1066] on J2 in 0.06s, 4 tests Suite: org.elasticsearch.common.util.concurrent.AbstractRunnableTests Completed [911/1066] on J1 in 0.06s, 6 tests Suite: org.elasticsearch.common.util.concurrent.AsyncIOProcessorTests Completed [912/1066] on J0 in 0.07s, 4 tests Suite: org.elasticsearch.common.util.concurrent.EsExecutorsTests Completed [913/1066] on J2 in 0.16s, 7 tests Suite: org.elasticsearch.common.util.concurrent.EsThreadPoolExecutorTests Completed [914/1066] on J1 in 0.14s, 1 test Suite: org.elasticsearch.common.util.concurrent.ThreadContextTests Completed [915/1066] on J2 in 0.07s, 19 tests Suite: org.elasticsearch.common.xcontent.smile.JsonVsSmileTests Completed [916/1066] on J1 in 0.01s, 1 test Suite: org.elasticsearch.common.util.concurrent.SizeBlockingQueueTests Completed [917/1066] on J0 in 0.45s, 1 test Suite: org.elasticsearch.common.xcontent.yaml.YamlXContentTests Completed [918/1066] on J2 in 0.13s, 50 tests Suite: org.elasticsearch.deps.jackson.JacksonLocationTests Completed [919/1066] on J1 in 0.02s, 1 test Suite: org.elasticsearch.discovery.zen.ZenPingTests Completed [920/1066] on J0 in 0.02s, 1 test Suite: org.elasticsearch.env.ESFileStoreTests Completed [921/1066] on J2 in 0.02s, 1 test Suite: org.elasticsearch.env.EnvironmentTests Completed [922/1066] on J1 in 0.16s, 11 tests Suite: org.elasticsearch.gateway.GatewayMetaStateTests Completed [923/1066] on J2 in 0.07s, 16 tests Suite: org.elasticsearch.index.IndexSettingsTests Completed [924/1066] on J1 in 0.26s, 21 tests Suite: org.elasticsearch.index.analysis.CharMatcherTests Completed [925/1066] on J2 in 0.02s, 5 tests Suite: org.elasticsearch.index.analysis.StopTokenFilterTests Completed [926/1066] on J2 in 0.06s, 3 tests Suite: org.elasticsearch.env.NodeEnvironmentTests Completed [927/1066] on J0 in 0.97s, 12 tests Suite: org.elasticsearch.index.analysis.CoreAnalysisFactoryTests Completed [928/1066] on J1 in 0.29s, 5 tests Suite: org.elasticsearch.index.analysis.WhitespaceTokenizerFactoryTests Completed [929/1066] on J2 in 0.03s, 2 tests Suite: org.elasticsearch.index.analysis.synonyms.SynonymsAnalysisTests Completed [930/1066] on J0 in 0.10s, 3 tests Suite: org.elasticsearch.index.fielddata.fieldcomparator.ReplaceMissingTests Completed [931/1066] on J2 in 0.02s, 1 test Suite: org.elasticsearch.index.fielddata.plain.HalfFloatFielddataTests Completed [932/1066] on J0 in 0.08s, 2 tests Suite: org.elasticsearch.index.mapper.BooleanFieldTypeTests Completed [933/1066] on J2 in 0.06s, 8 tests Suite: org.elasticsearch.index.mapper.DynamicTemplateTests Completed [934/1066] on J2 in 0.12s, 6 tests Suite: org.elasticsearch.index.mapper.DynamicMappingVersionTests Completed [935/1066] on J0 in 0.44s, 2 tests Suite: org.elasticsearch.index.mapper.ExternalFieldMapperTests Completed [936/1066] on J2 in 0.69s, 3 tests Suite: org.elasticsearch.index.mapper.FieldNamesFieldMapperTests Completed [937/1066] on J2 in 1.06s, 6 tests Suite: org.elasticsearch.index.mapper.FieldFilterMapperPluginTests Completed [938/1066] on J0 in 2.30s, 4 tests Suite: org.elasticsearch.index.fielddata.NoOrdinalsStringFieldDataTests IGNOR/A 0.10s J1 | NoOrdinalsStringFieldDataTests.testTermsEnum > Assumption #1: We can't test this, since the returned IFD instance doesn't implement IndexFieldData.WithOrdinals Completed [939/1066] on J1 in 3.30s, 19 tests, 1 skipped Suite: org.elasticsearch.index.mapper.GeoShapeFieldTypeTests Completed [940/1066] on J0 in 0.05s, 6 tests Suite: org.elasticsearch.index.mapper.IndexFieldTypeTests Completed [941/1066] on J0 in 0.04s, 5 tests Suite: org.elasticsearch.index.mapper.IdFieldMapperTests Completed [942/1066] on J1 in 0.31s, 2 tests Suite: org.elasticsearch.index.mapper.NullValueObjectMappingTests Completed [943/1066] on J0 in 0.21s, 1 test Suite: org.elasticsearch.index.mapper.TypeFieldTypeTests Completed [944/1066] on J1 in 0.05s, 6 tests Suite: org.elasticsearch.index.mapper.GeoShapeFieldMapperTests Completed [945/1066] on J2 in 1.48s, 12 tests Suite: org.elasticsearch.index.mapper.UidTests Completed [946/1066] on J0 in 0.16s, 4 tests Suite: org.elasticsearch.index.query.BoostingQueryBuilderTests Completed [947/1066] on J2 in 0.12s, 12 tests Suite: org.elasticsearch.index.query.CombineFunctionTests Completed [948/1066] on J0 in 0.05s, 4 tests Suite: org.elasticsearch.index.query.ExistsQueryBuilderTests Completed [949/1066] on J0 in 0.11s, 11 tests Suite: org.elasticsearch.index.mapper.UpdateMappingTests Completed [950/1066] on J1 in 0.69s, 4 tests Suite: org.elasticsearch.index.query.FuzzyQueryBuilderTests Completed [951/1066] on J0 in 0.14s, 19 tests Suite: org.elasticsearch.index.query.GeoBoundingBoxQueryBuilderTests IGNOR/A 0.00s J1 | GeoBoundingBoxQueryBuilderTests.testExceptionOnMissingTypes > Assumption #1: test runs only when at least a type is registered Completed [952/1066] on J1 in 0.23s, 29 tests, 1 skipped Suite: org.elasticsearch.index.query.ScriptQueryBuilderTests Completed [953/1066] on J0 in 0.17s, 13 tests Suite: org.elasticsearch.index.query.DisableGraphQueryTests Completed [954/1066] on J2 in 0.90s, 5 tests Suite: org.elasticsearch.index.query.SpanOrQueryBuilderTests Completed [955/1066] on J0 in 0.15s, 12 tests Suite: org.elasticsearch.index.query.SimpleQueryStringBuilderTests IGNOR/A 0.00s J1 | SimpleQueryStringBuilderTests.testAnalyzerWithGraph > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.00s J1 | SimpleQueryStringBuilderTests.testExpandedTerms > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.07s J1 | SimpleQueryStringBuilderTests.testDefaultField > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.00s J1 | SimpleQueryStringBuilderTests.testToQueryBoost > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.00s J1 | SimpleQueryStringBuilderTests.testLenientToPrefixQuery > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.00s J1 | SimpleQueryStringBuilderTests.testToFuzzyQuery > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.00s J1 | SimpleQueryStringBuilderTests.testMinimumShouldMatch > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.00s J1 | SimpleQueryStringBuilderTests.testAnalyzeWildcard > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.00s J1 | SimpleQueryStringBuilderTests.testQuoteFieldSuffix > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.00s J1 | SimpleQueryStringBuilderTests.testWithStopWords > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.00s J1 | SimpleQueryStringBuilderTests.testWithPrefixStopWords > Assumption #1: test runs only when at least a type is registered IGNOR/A 0.00s J1 | SimpleQueryStringBuilderTests.testAnalyzerWildcardWithSynonyms > Assumption #1: test runs only when at least a type is registered Completed [956/1066] on J1 in 0.40s, 36 tests, 12 skipped Suite: org.elasticsearch.index.query.TypeQueryBuilderTests Completed [957/1066] on J2 in 0.14s, 11 tests Suite: org.elasticsearch.index.query.WildcardQueryBuilderTests Completed [958/1066] on J0 in 0.13s, 15 tests Suite: org.elasticsearch.index.query.WrapperQueryBuilderTests Completed [959/1066] on J1 in 0.12s, 14 tests Suite: org.elasticsearch.index.query.functionscore.ScoreFunctionBuilderTests Completed [960/1066] on J2 in 0.03s, 2 tests Suite: org.elasticsearch.index.refresh.RefreshStatsTests Completed [961/1066] on J0 in 0.02s, 2 tests Suite: org.elasticsearch.index.reindex.BulkByScrollResponseTests Completed [962/1066] on J1 in 0.02s, 1 test Suite: org.elasticsearch.index.search.stats.SearchStatsTests Completed [963/1066] on J0 in 0.02s, 1 test Suite: org.elasticsearch.index.seqno.CountedBitSetTests Completed [964/1066] on J1 in 0.03s, 2 tests Suite: org.elasticsearch.index.search.nested.NestedSortingTests Completed [965/1066] on J2 in 0.26s, 3 tests Suite: org.elasticsearch.index.seqno.ReplicationTrackerTests Completed [966/1066] on J0 in 0.12s, 12 tests Suite: org.elasticsearch.index.shard.IndexShardOperationPermitsTests Completed [967/1066] on J1 in 0.13s, 14 tests Suite: org.elasticsearch.indices.IndicesModuleTests Completed [968/1066] on J0 in 0.05s, 13 tests Suite: org.elasticsearch.indices.IndicesQueryCacheTests Completed [969/1066] on J1 in 0.08s, 4 tests Suite: org.elasticsearch.indices.NodeIndicesStatsTests Completed [970/1066] on J1 in 0.02s, 1 test Suite: org.elasticsearch.ingest.PipelineStoreTests Completed [971/1066] on J1 in 0.05s, 10 tests Suite: org.elasticsearch.monitor.fs.FsProbeTests Completed [972/1066] on J1 in 0.07s, 4 tests Suite: org.elasticsearch.monitor.os.OsProbeTests IGNOR/A 0.00s J1 | OsProbeTests.testCgroupProbe > Assumption #1: test runs on Linux only IGNOR/A 0.00s J1 | OsProbeTests.testGetSystemLoadAverage > Assumption #1: test runs on Linux only Completed [973/1066] on J1 in 0.03s, 4 tests, 2 skipped Suite: org.elasticsearch.node.NodeTests Completed [974/1066] on J1 in 0.31s, 4 tests Suite: org.elasticsearch.node.ResponseCollectorServiceTests Completed [975/1066] on J1 in 0.03s, 3 tests Suite: org.elasticsearch.persistent.CancelPersistentTaskRequestTests Completed [976/1066] on J1 in 0.03s, 2 tests Suite: org.elasticsearch.persistent.PersistentTasksClusterServiceTests Completed [977/1066] on J1 in 0.09s, 10 tests Suite: org.elasticsearch.persistent.PersistentTasksExecutorResponseTests Completed [978/1066] on J1 in 0.21s, 2 tests Suite: org.elasticsearch.persistent.PersistentTasksNodeServiceStatusTests Completed [979/1066] on J1 in 0.06s, 3 tests Suite: org.elasticsearch.persistent.PersistentTasksNodeServiceTests Completed [980/1066] on J1 in 0.36s, 3 tests Suite: org.elasticsearch.persistent.RestartPersistentTaskRequestTests Completed [981/1066] on J1 in 0.22s, 2 tests Suite: org.elasticsearch.persistent.StartPersistentActionRequestTests Completed [982/1066] on J1 in 0.16s, 2 tests Suite: org.elasticsearch.persistent.UpdatePersistentTaskRequestTests Completed [983/1066] on J1 in 0.07s, 2 tests Suite: org.elasticsearch.indices.IndicesServiceTests Completed [984/1066] on J0 in 4.81s, 9 tests Suite: org.elasticsearch.plugins.spi.NamedXContentProviderTests Completed [985/1066] on J0 in 0.07s, 2 tests Suite: org.elasticsearch.plugins.MetaPluginInfoTests Completed [986/1066] on J1 in 0.27s, 5 tests Suite: org.elasticsearch.rest.action.admin.indices.RestAnalyzeActionTests Completed [987/1066] on J1 in 0.05s, 7 tests Suite: org.elasticsearch.recovery.RecoveriesCollectionTests Completed [988/1066] on J0 in 0.48s, 4 tests Suite: org.elasticsearch.rest.action.admin.indices.RestResizeHandlerTests Completed [989/1066] on J1 in 0.02s, 2 tests Suite: org.elasticsearch.rest.action.cat.RestIndicesActionTests Completed [990/1066] on J0 in 0.04s, 1 test Suite: org.elasticsearch.rest.action.cat.RestNodesActionTests Completed [991/1066] on J1 in 0.03s, 1 test Suite: org.elasticsearch.rest.action.cat.RestRecoveryActionTests Completed [992/1066] on J0 in 0.09s, 1 test Suite: org.elasticsearch.search.MultiValueModeTests Completed [993/1066] on J1 in 0.04s, 12 tests Suite: org.elasticsearch.search.SearchHitTests Completed [994/1066] on J0 in 0.96s, 8 tests Suite: org.elasticsearch.search.SearchSortValuesTests Completed [995/1066] on J0 in 0.03s, 4 tests Suite: org.elasticsearch.search.aggregations.MultiBucketAggregatorWrapperTests Completed [996/1066] on J0 in 0.04s, 1 test Suite: org.elasticsearch.search.aggregations.SubAggCollectionModeTests Completed [997/1066] on J0 in 0.02s, 4 tests Suite: org.elasticsearch.search.aggregations.bucket.BestBucketsDeferringCollectorTests Completed [998/1066] on J0 in 0.04s, 1 test Suite: org.elasticsearch.search.aggregations.bucket.DateRangeTests Completed [999/1066] on J0 in 0.06s, 6 tests Suite: org.elasticsearch.search.aggregations.bucket.GeoHashGridTests Completed [1000/1066] on J0 in 0.45s, 5 tests Suite: org.elasticsearch.search.aggregations.bucket.GlobalAggregatorTests Completed [1001/1066] on J0 in 0.03s, 2 tests Suite: org.elasticsearch.search.aggregations.bucket.HistogramTests Completed [1002/1066] on J0 in 0.06s, 6 tests Suite: org.elasticsearch.search.aggregations.bucket.IpRangeTests Completed [1003/1066] on J0 in 0.11s, 6 tests Suite: org.elasticsearch.search.aggregations.bucket.RangeTests Completed [1004/1066] on J0 in 0.11s, 7 tests Suite: org.elasticsearch.search.aggregations.bucket.SamplerTests Completed [1005/1066] on J0 in 0.04s, 5 tests Suite: org.elasticsearch.search.SearchServiceTests Completed [1006/1066] on J1 in 6.73s, 10 tests Suite: org.elasticsearch.search.aggregations.bucket.composite.CompositeValuesCollectorQueueTests Completed [1007/1066] on J1 in 1.85s, 10 tests Suite: org.elasticsearch.search.aggregations.bucket.composite.InternalCompositeTests Completed [1008/1066] on J1 in 0.29s, 7 tests Suite: org.elasticsearch.search.aggregations.bucket.filter.InternalFilterTests Completed [1009/1066] on J1 in 0.10s, 5 tests Suite: org.elasticsearch.search.aggregations.bucket.filter.InternalFiltersTests Completed [1010/1066] on J1 in 0.15s, 6 tests Suite: org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregatorTests Completed [1011/1066] on J0 in 5.64s, 26 tests Suite: org.elasticsearch.search.aggregations.bucket.geogrid.GeoHashGridAggregatorTests Completed [1012/1066] on J1 in 0.07s, 3 tests Suite: org.elasticsearch.search.aggregations.bucket.histogram.InternalDateHistogramTests Completed [1013/1066] on J0 in 0.15s, 6 tests Suite: org.elasticsearch.search.aggregations.bucket.nested.InternalNestedTests Completed [1014/1066] on J1 in 0.05s, 5 tests Suite: org.elasticsearch.search.aggregations.bucket.nested.InternalReverseNestedTests Completed [1015/1066] on J0 in 0.05s, 5 tests Suite: org.elasticsearch.search.aggregations.bucket.nested.NestedAggregatorTests Completed [1016/1066] on J1 in 0.49s, 8 tests Suite: org.elasticsearch.search.aggregations.bucket.nested.NestedTests Completed [1017/1066] on J0 in 0.30s, 5 tests Suite: org.elasticsearch.search.aggregations.bucket.nested.ReverseNestedAggregatorTests Completed [1018/1066] on J1 in 0.31s, 2 tests Suite: org.elasticsearch.search.aggregations.bucket.range.InternalDateRangeTests Completed [1019/1066] on J0 in 0.41s, 6 tests Suite: org.elasticsearch.search.aggregations.bucket.significant.SignificantTermsAggregatorTests Completed [1020/1066] on J1 in 0.16s, 4 tests Suite: org.elasticsearch.search.aggregations.metrics.GeoCentroidTests Completed [1021/1066] on J1 in 0.07s, 5 tests Suite: org.elasticsearch.search.aggregations.bucket.terms.LongTermsTests Completed [1022/1066] on J0 in 0.52s, 6 tests Suite: org.elasticsearch.search.aggregations.metrics.InternalExtendedStatsTests Completed [1023/1066] on J1 in 0.28s, 6 tests Suite: org.elasticsearch.search.aggregations.metrics.InternalStatsTests Completed [1024/1066] on J0 in 0.07s, 7 tests Suite: org.elasticsearch.search.aggregations.metrics.InternalSumTests Completed [1025/1066] on J1 in 0.08s, 6 tests Suite: org.elasticsearch.index.shard.IndexShardTests Completed [1026/1066] on J2 in 19.96s, 57 tests Suite: org.elasticsearch.search.aggregations.metrics.MaxAggregatorTests Completed [1027/1066] on J0 in 0.07s, 6 tests Suite: org.elasticsearch.search.aggregations.metrics.avg.AvgAggregatorTests Completed [1028/1066] on J1 in 0.11s, 7 tests Suite: org.elasticsearch.search.aggregations.metrics.percentiles.PercentilesMethodTests Completed [1029/1066] on J2 in 0.05s, 4 tests Suite: org.elasticsearch.search.aggregations.metrics.scripted.InternalScriptedMetricTests Completed [1030/1066] on J0 in 0.05s, 5 tests Suite: org.elasticsearch.search.aggregations.metrics.valuecount.InternalValueCountTests Completed [1031/1066] on J1 in 0.08s, 5 tests Suite: org.elasticsearch.search.aggregations.metrics.valuecount.ValueCountAggregatorTests Completed [1032/1066] on J2 in 0.09s, 6 tests Suite: org.elasticsearch.search.aggregations.pipeline.BucketScriptTests Completed [1033/1066] on J0 in 0.05s, 3 tests Suite: org.elasticsearch.search.aggregations.pipeline.BucketSelectorTests Completed [1034/1066] on J1 in 0.03s, 3 tests Suite: org.elasticsearch.search.aggregations.pipeline.CumulativeSumAggregatorTests Completed [1035/1066] on J2 in 0.16s, 5 tests Suite: org.elasticsearch.search.aggregations.pipeline.CumulativeSumTests Completed [1036/1066] on J0 in 0.02s, 3 tests Suite: org.elasticsearch.search.aggregations.pipeline.DerivativeTests Completed [1037/1066] on J1 in 0.03s, 3 tests Suite: org.elasticsearch.search.aggregations.pipeline.InternalSimpleValueTests Completed [1038/1066] on J2 in 0.03s, 5 tests Suite: org.elasticsearch.search.aggregations.pipeline.bucketmetrics.ExtendedStatsBucketTests Completed [1039/1066] on J0 in 0.04s, 5 tests Suite: org.elasticsearch.search.builder.SearchSourceBuilderTests Completed [1040/1066] on J2 in 0.20s, 16 tests Suite: org.elasticsearch.search.profile.query.QueryProfileShardResultTests Completed [1041/1066] on J2 in 0.03s, 1 test Suite: org.elasticsearch.search.collapse.CollapseBuilderTests Completed [1042/1066] on J0 in 0.27s, 5 tests Suite: org.elasticsearch.search.aggregations.support.ValuesSourceConfigTests Completed [1043/1066] on J1 in 0.89s, 9 tests Suite: org.elasticsearch.search.profile.query.QueryProfilerTests Completed [1044/1066] on J2 in 0.06s, 6 tests Suite: org.elasticsearch.search.slice.DocValuesSliceQueryTests Completed [1045/1066] on J0 in 0.06s, 2 tests Suite: org.elasticsearch.search.slice.TermsSliceQueryTests Completed [1046/1066] on J0 in 0.04s, 2 tests Suite: org.elasticsearch.search.sort.GeoDistanceSortBuilderTests Completed [1047/1066] on J2 in 0.08s, 18 tests Suite: org.elasticsearch.search.slice.SliceBuilderTests Completed [1048/1066] on J1 in 0.75s, 10 tests Suite: org.elasticsearch.search.sort.ScriptSortBuilderTests Completed [1049/1066] on J2 in 0.29s, 19 tests Suite: org.elasticsearch.search.sort.SortBuilderTests Completed [1050/1066] on J1 in 0.12s, 3 tests Suite: org.elasticsearch.search.sort.NestedSortBuilderTests Completed [1051/1066] on J0 in 0.89s, 4 tests Suite: org.elasticsearch.search.suggest.TermSuggestionOptionTests Completed [1052/1066] on J1 in 0.04s, 3 tests Suite: org.elasticsearch.search.suggest.SuggestionTests Completed [1053/1066] on J2 in 0.25s, 5 tests Suite: org.elasticsearch.snapshots.SnapshotUtilsTests Completed [1054/1066] on J0 in 0.02s, 1 test Suite: org.elasticsearch.snapshots.SnapshotsInProgressSerializationTests Completed [1055/1066] on J1 in 0.06s, 3 tests Suite: org.elasticsearch.tasks.ListTasksResponseTests Completed [1056/1066] on J2 in 0.02s, 2 tests Suite: org.elasticsearch.tasks.TaskIdTests Completed [1057/1066] on J0 in 0.02s, 2 tests Suite: org.elasticsearch.tasks.TaskResultTests Completed [1058/1066] on J1 in 0.03s, 3 tests Suite: org.elasticsearch.threadpool.AutoQueueAdjustingExecutorBuilderTests Completed [1059/1066] on J2 in 0.02s, 1 test Suite: org.elasticsearch.threadpool.UpdateThreadPoolSettingsTests Completed [1060/1066] on J0 in 0.04s, 6 tests Suite: org.elasticsearch.transport.CompressibleBytesOutputStreamTests Completed [1061/1066] on J1 in 0.02s, 3 tests Suite: org.elasticsearch.transport.ConnectionProfileTests Completed [1062/1066] on J2 in 0.02s, 2 tests Suite: org.elasticsearch.transport.RemoteClusterClientTests IGNOR/A 0.00s J0 | RemoteClusterClientTests.testEnsureWeReconnect > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29547)) Completed [1063/1066] on J0 in 0.12s, 2 tests, 1 skipped Suite: org.elasticsearch.usage.UsageServiceTests Completed [1064/1066] on J2 in 0.02s, 1 test Suite: org.elasticsearch.watcher.FileWatcherTests Completed [1065/1066] on J0 in 0.14s, 7 tests Suite: org.elasticsearch.transport.RemoteClusterConnectionTests Completed [1066/1066] on J1 in 4.58s, 19 tests > Task :server:test Slow Tests Summary: 26.64s | org.elasticsearch.index.engine.InternalEngineTests 19.96s | org.elasticsearch.index.shard.IndexShardTests 17.10s | org.elasticsearch.index.shard.RefreshListenersTests 16.31s | org.elasticsearch.index.fielddata.ScriptDocValuesLongsTests 14.97s | org.elasticsearch.transport.RemoteClusterServiceTests ==> Test Summary: 1066 suites (1 ignored), 6908 tests, 50 ignored (50 assumptions) [ant:junit4] JVM J0: 0.66 .. 236.77 = 236.11s [ant:junit4] JVM J1: 0.92 .. 241.05 = 240.12s [ant:junit4] JVM J2: 0.60 .. 236.53 = 235.93s [ant:junit4] Execution time total: 4 minutes 1 second [ant:junit4] Tests summary: 1066 suites (1 ignored), 6908 tests, 50 ignored (50 assumptions) :server:test (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 4 mins 1.387 secs. :benchmarks:test (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:test SKIPPED Skipping task ':benchmarks:test' as task onlyIf is false. :benchmarks:test (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :benchmarks:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:jar Task ':benchmarks:jar' is not up-to-date because: No history is available. :benchmarks:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.026 secs. :benchmarks:extractJar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:extractJar Task ':benchmarks:extractJar' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\jar''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\jar-extracted\jar Command: C:\Users\jenkins\.java\java10\bin\jar xf C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\distributions\elasticsearch-benchmarks-7.0.0-alpha1-SNAPSHOT.jar META-INF/LICENSE.txt META-INF/NOTICE.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\jar'' :benchmarks:extractJar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.213 secs. :benchmarks:verifyJarLicense (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:verifyJarLicense Task ':benchmarks:verifyJarLicense' is not up-to-date because: Task has not declared any outputs despite executing actions. :benchmarks:verifyJarLicense (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.002 secs. :benchmarks:verifyJarNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:verifyJarNotice Task ':benchmarks:verifyJarNotice' is not up-to-date because: Task has not declared any outputs despite executing actions. :benchmarks:verifyJarNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :libs:elasticsearch-core:javadoc (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:elasticsearch-core:javadoc Task ':libs:elasticsearch-core:javadoc' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javadoc''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core Command: C:\Users\jenkins\.java\java10\bin\javadoc @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\tmp\javadoc\javadoc.options Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javadoc'' javadoc: warning - You have not specified the version of HTML to use. The default is currently HTML 4.01, but this will change to HTML5 in a future release. To suppress this warning, please specify the version of HTML used in your documentation comments and to be generated by this doclet, using the -html4 or -html5 options. 1 warning :libs:elasticsearch-core:javadoc (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.156 secs. :libs:plugin-classloader:javadoc (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:plugin-classloader:javadoc Task ':libs:plugin-classloader:javadoc' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javadoc''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader Command: C:\Users\jenkins\.java\java10\bin\javadoc @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\plugin-classloader\build\tmp\javadoc\javadoc.options Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javadoc'' javadoc: warning - You have not specified the version of HTML to use. The default is currently HTML 4.01, but this will change to HTML5 in a future release. To suppress this warning, please specify the version of HTML used in your documentation comments and to be generated by this doclet, using the -html4 or -html5 options. 1 warning :libs:plugin-classloader:javadoc (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.522 secs. :libs:secure-sm:javadoc (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:secure-sm:javadoc Task ':libs:secure-sm:javadoc' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javadoc''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm Command: C:\Users\jenkins\.java\java10\bin\javadoc @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\tmp\javadoc\javadoc.options Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javadoc'' javadoc: warning - You have not specified the version of HTML to use. The default is currently HTML 4.01, but this will change to HTML5 in a future release. To suppress this warning, please specify the version of HTML used in your documentation comments and to be generated by this doclet, using the -html4 or -html5 options. 1 warning :libs:secure-sm:javadoc (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.527 secs. :libs:x-content:javadoc (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:x-content:javadoc Task ':libs:x-content:javadoc' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javadoc''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content Command: C:\Users\jenkins\.java\java10\bin\javadoc @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\tmp\javadoc\javadoc.options Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javadoc'' javadoc: warning - You have not specified the version of HTML to use. The default is currently HTML 4.01, but this will change to HTML5 in a future release. To suppress this warning, please specify the version of HTML used in your documentation comments and to be generated by this doclet, using the -html4 or -html5 options. 1 warning :libs:x-content:javadoc (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 3.873 secs. :server:cli:javadoc (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:cli:javadoc Task ':server:cli:javadoc' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javadoc''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli Command: C:\Users\jenkins\.java\java10\bin\javadoc @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\tmp\javadoc\javadoc.options Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javadoc'' javadoc: warning - You have not specified the version of HTML to use. The default is currently HTML 4.01, but this will change to HTML5 in a future release. To suppress this warning, please specify the version of HTML used in your documentation comments and to be generated by this doclet, using the -html4 or -html5 options. 1 warning :server:cli:javadoc (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.875 secs. :server:javadoc (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :server:javadoc Task ':server:javadoc' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javadoc''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server Command: C:\Users\jenkins\.java\java10\bin\javadoc @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\tmp\javadoc\javadoc.options Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javadoc'' javadoc: warning - You have not specified the version of HTML to use. The default is currently HTML 4.01, but this will change to HTML5 in a future release. To suppress this warning, please specify the version of HTML used in your documentation comments and to be generated by this doclet, using the -html4 or -html5 options. 1 warning :server:javadoc (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 47.876 secs. :benchmarks:javadoc (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:javadoc Task ':benchmarks:javadoc' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javadoc''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks Command: C:\Users\jenkins\.java\java10\bin\javadoc @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\tmp\javadoc\javadoc.options Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javadoc'' javadoc: warning - You have not specified the version of HTML to use. The default is currently HTML 4.01, but this will change to HTML5 in a future release. To suppress this warning, please specify the version of HTML used in your documentation comments and to be generated by this doclet, using the -html4 or -html5 options. 1 warning :benchmarks:javadoc (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.054 secs. :benchmarks:javadocJar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:javadocJar Task ':benchmarks:javadocJar' is not up-to-date because: No history is available. :benchmarks:javadocJar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.094 secs. :benchmarks:extractJavadocJar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:extractJavadocJar Task ':benchmarks:extractJavadocJar' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\jar''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\jar-extracted\javadocJar Command: C:\Users\jenkins\.java\java10\bin\jar xf C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\distributions\elasticsearch-benchmarks-7.0.0-alpha1-SNAPSHOT-javadoc.jar META-INF/LICENSE.txt META-INF/NOTICE.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\jar'' :benchmarks:extractJavadocJar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.215 secs. :benchmarks:verifyJavadocJarLicense (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:verifyJavadocJarLicense Task ':benchmarks:verifyJavadocJarLicense' is not up-to-date because: Task has not declared any outputs despite executing actions. :benchmarks:verifyJavadocJarLicense (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :benchmarks:verifyJavadocJarNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:verifyJavadocJarNotice Task ':benchmarks:verifyJavadocJarNotice' is not up-to-date because: Task has not declared any outputs despite executing actions. :benchmarks:verifyJavadocJarNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :benchmarks:shadowJar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:shadowJar Task ':benchmarks:shadowJar' is not up-to-date because: No history is available. ******************* GRADLE SHADOW STATS Total Jars: 40 (includes project) Total Time: 4.485s [4485ms] Average Time/Jar: 0.112125s [112.125ms] ******************* :benchmarks:shadowJar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 4.772 secs. :benchmarks:extractShadowJar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:extractShadowJar Task ':benchmarks:extractShadowJar' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\jar''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\jar-extracted\shadowJar Command: C:\Users\jenkins\.java\java10\bin\jar xf C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\distributions\elasticsearch-benchmarks-7.0.0-alpha1-SNAPSHOT-benchmarks.jar META-INF/LICENSE.txt META-INF/NOTICE.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\jar'' :benchmarks:extractShadowJar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.252 secs. :benchmarks:verifyShadowJarLicense (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:verifyShadowJarLicense Task ':benchmarks:verifyShadowJarLicense' is not up-to-date because: Task has not declared any outputs despite executing actions. :benchmarks:verifyShadowJarLicense (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :benchmarks:verifyShadowJarNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:verifyShadowJarNotice Task ':benchmarks:verifyShadowJarNotice' is not up-to-date because: Task has not declared any outputs despite executing actions. :benchmarks:verifyShadowJarNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :benchmarks:sourcesJar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:sourcesJar Task ':benchmarks:sourcesJar' is not up-to-date because: No history is available. :benchmarks:sourcesJar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.015 secs. :benchmarks:extractSourcesJar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:extractSourcesJar Task ':benchmarks:extractSourcesJar' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\jar''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\jar-extracted\sourcesJar Command: C:\Users\jenkins\.java\java10\bin\jar xf C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\benchmarks\build\distributions\elasticsearch-benchmarks-7.0.0-alpha1-SNAPSHOT-sources.jar META-INF/LICENSE.txt META-INF/NOTICE.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\jar'' :benchmarks:extractSourcesJar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.198 secs. :benchmarks:verifySourcesJarLicense (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:verifySourcesJarLicense Task ':benchmarks:verifySourcesJarLicense' is not up-to-date because: Task has not declared any outputs despite executing actions. :benchmarks:verifySourcesJarLicense (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :benchmarks:verifySourcesJarNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:verifySourcesJarNotice Task ':benchmarks:verifySourcesJarNotice' is not up-to-date because: Task has not declared any outputs despite executing actions. :benchmarks:verifySourcesJarNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :benchmarks:check (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:check Skipping task ':benchmarks:check' as it has no actions. :benchmarks:check (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :benchmarks:build (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :benchmarks:build Skipping task ':benchmarks:build' as it has no actions. :benchmarks:build (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :build-tools:generatePomFileForNebulaPublication (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :build-tools:generatePomFileForNebulaPublication Task ':build-tools:generatePomFileForNebulaPublication' is not up-to-date because: Task.upToDateWhen is false. :build-tools:generatePomFileForNebulaPublication (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.156 secs. :build-tools:javadoc (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :build-tools:javadoc Task ':build-tools:javadoc' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javadoc''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc Command: C:\Users\jenkins\.java\java10\bin\javadoc @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\tmp\javadoc\javadoc.options Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javadoc'' javadoc: warning - You have not specified the version of HTML to use. The default is currently HTML 4.01, but this will change to HTML5 in a future release. To suppress this warning, please specify the version of HTML used in your documentation comments and to be generated by this doclet, using the -html4 or -html5 options. 1 warning :build-tools:javadoc (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.257 secs. :build-tools:javadocJar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :build-tools:javadocJar Task ':build-tools:javadocJar' is not up-to-date because: No history is available. :build-tools:javadocJar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.094 secs. :build-tools:sourcesJar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :build-tools:sourcesJar Task ':build-tools:sourcesJar' is not up-to-date because: No history is available. :build-tools:sourcesJar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.052 secs. :build-tools:assemble (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :build-tools:assemble Skipping task ':build-tools:assemble' as it has no actions. :build-tools:assemble (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :build-tools:extractJar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :build-tools:extractJar Task ':build-tools:extractJar' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\jar''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\jar-extracted\jar Command: C:\Users\jenkins\.java\java10\bin\jar xf C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\distributions\build-tools-7.0.0-alpha1-SNAPSHOT.jar META-INF/LICENSE.txt META-INF/NOTICE.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\jar'' :build-tools:extractJar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.259 secs. :build-tools:verifyJarLicense (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :build-tools:verifyJarLicense Task ':build-tools:verifyJarLicense' is not up-to-date because: Task has not declared any outputs despite executing actions. :build-tools:verifyJarLicense (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.003 secs. :build-tools:verifyJarNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :build-tools:verifyJarNotice Task ':build-tools:verifyJarNotice' is not up-to-date because: Task has not declared any outputs despite executing actions. :build-tools:verifyJarNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :build-tools:extractJavadocJar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :build-tools:extractJavadocJar Task ':build-tools:extractJavadocJar' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\jar''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\jar-extracted\javadocJar Command: C:\Users\jenkins\.java\java10\bin\jar xf C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\distributions\build-tools-7.0.0-alpha1-SNAPSHOT-javadoc.jar META-INF/LICENSE.txt META-INF/NOTICE.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\jar'' :build-tools:extractJavadocJar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.329 secs. :build-tools:verifyJavadocJarLicense (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :build-tools:verifyJavadocJarLicense Task ':build-tools:verifyJavadocJarLicense' is not up-to-date because: Task has not declared any outputs despite executing actions. :build-tools:verifyJavadocJarLicense (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :build-tools:verifyJavadocJarNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :build-tools:verifyJavadocJarNotice Task ':build-tools:verifyJavadocJarNotice' is not up-to-date because: Task has not declared any outputs despite executing actions. :build-tools:verifyJavadocJarNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :build-tools:extractSourcesJar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :build-tools:extractSourcesJar Task ':build-tools:extractSourcesJar' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\jar''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\jar-extracted\sourcesJar Command: C:\Users\jenkins\.java\java10\bin\jar xf C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\distributions\build-tools-7.0.0-alpha1-SNAPSHOT-sources.jar META-INF/LICENSE.txt META-INF/NOTICE.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\jar'' :build-tools:extractSourcesJar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.195 secs. :build-tools:verifySourcesJarLicense (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :build-tools:verifySourcesJarLicense Task ':build-tools:verifySourcesJarLicense' is not up-to-date because: Task has not declared any outputs despite executing actions. :build-tools:verifySourcesJarLicense (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :build-tools:verifySourcesJarNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :build-tools:verifySourcesJarNotice Task ':build-tools:verifySourcesJarNotice' is not up-to-date because: Task has not declared any outputs despite executing actions. :build-tools:verifySourcesJarNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :build-tools:check (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :build-tools:check Skipping task ':build-tools:check' as it has no actions. :build-tools:check (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :build-tools:build (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :build-tools:build Skipping task ':build-tools:build' as it has no actions. :build-tools:build (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :docs:buildRestTests (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:buildRestTests Task ':docs:buildRestTests' is not up-to-date because: No history is available. :docs:buildRestTests (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.726 secs. :docs:copyCheckstyleConf (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:copyCheckstyleConf Task ':docs:copyCheckstyleConf' is not up-to-date because: No history is available. :docs:copyCheckstyleConf (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.004 secs. :docs:compileTestJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:compileTestJava Task ':docs:compileTestJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':docs:compileTestJava'. Compiling with JDK Java compiler API. :docs:compileTestJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.649 secs. :docs:processTestResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:processTestResources Task ':docs:processTestResources' is not up-to-date because: No history is available. :docs:processTestResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.004 secs. :docs:testClasses (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:testClasses Skipping task ':docs:testClasses' as it has no actions. :docs:testClasses (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :docs:checkstyleTest (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:checkstyleTest Task ':docs:checkstyleTest' is not up-to-date because: No history is available. [ant:checkstyle] Running Checkstyle 7.5 on 1 files :docs:checkstyleTest (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.072 secs. :docs:forbiddenApisTest (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:forbiddenApisTest Task ':docs:forbiddenApisTest' is not up-to-date because: No history is available. Reading bundled API signatures: jdk-unsafe-1.8 Reading bundled API signatures: jdk-deprecated-1.8 Reading bundled API signatures: jdk-non-portable Reading bundled API signatures: jdk-system-out Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/jdk-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-all-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/es-test-signatures.txt Reading API signatures: jar:file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/jars-3/948350e590c59730d998fca29541ce1f/buildSrc-7.0.0-alpha1-SNAPSHOT.jar!/forbidden/http-signatures.txt Loading classes to check... Scanning classes for violations... Scanned 1 class file(s) for forbidden API invocations (in 0.04s), 0 error(s). :docs:forbiddenApisTest (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.05 secs. :docs:forbiddenApis (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:forbiddenApis Skipping task ':docs:forbiddenApis' as it has no actions. :docs:forbiddenApis (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :docs:integTestCluster#prepareCluster.cleanShared (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#prepareCluster.cleanShared Task ':docs:integTestCluster#prepareCluster.cleanShared' is not up-to-date because: Task has not declared any outputs despite executing actions. :docs:integTestCluster#prepareCluster.cleanShared (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :docs:integTestCluster#clean (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#clean UP-TO-DATE Task ':docs:integTestCluster#clean' is not up-to-date because: Task has not declared any outputs despite executing actions. :docs:integTestCluster#clean (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :docs:integTestCluster#createCwd (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#createCwd Task ':docs:integTestCluster#createCwd' is not up-to-date because: No history is available. :docs:integTestCluster#createCwd (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.002 secs. :docs:integTestCluster#checkPrevious (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#checkPrevious SKIPPED Skipping task ':docs:integTestCluster#checkPrevious' as task onlyIf is false. :docs:integTestCluster#checkPrevious (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :docs:integTestCluster#stopPrevious (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#stopPrevious SKIPPED Skipping task ':docs:integTestCluster#stopPrevious' as task onlyIf is false. :docs:integTestCluster#stopPrevious (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :distribution:buildDefaultNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:buildDefaultNotice Task ':distribution:buildDefaultNotice' is not up-to-date because: No history is available. :distribution:buildDefaultNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.041 secs. :modules:aggs-matrix-stats:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:aggs-matrix-stats:compileJava Task ':modules:aggs-matrix-stats:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':modules:aggs-matrix-stats:compileJava'. :modules:aggs-matrix-stats:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\aggs-matrix-stats Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\aggs-matrix-stats\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.015 secs. Created jar classpath snapshot for incremental compilation in 0.004 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :modules:aggs-matrix-stats:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.278 secs. :modules:aggs-matrix-stats:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:aggs-matrix-stats:processResources Task ':modules:aggs-matrix-stats:processResources' is not up-to-date because: No history is available. :modules:aggs-matrix-stats:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.005 secs. :modules:aggs-matrix-stats:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:aggs-matrix-stats:classes Skipping task ':modules:aggs-matrix-stats:classes' as it has no actions. :modules:aggs-matrix-stats:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:aggs-matrix-stats:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:aggs-matrix-stats:jar Task ':modules:aggs-matrix-stats:jar' is not up-to-date because: No history is available. :modules:aggs-matrix-stats:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.037 secs. :modules:aggs-matrix-stats:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:aggs-matrix-stats:copyPluginPropertiesTemplate Task ':modules:aggs-matrix-stats:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :modules:aggs-matrix-stats:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.002 secs. :modules:aggs-matrix-stats:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:aggs-matrix-stats:pluginProperties Task ':modules:aggs-matrix-stats:pluginProperties' is not up-to-date because: No history is available. :modules:aggs-matrix-stats:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.032 secs. :modules:aggs-matrix-stats:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:aggs-matrix-stats:bundlePlugin Task ':modules:aggs-matrix-stats:bundlePlugin' is not up-to-date because: No history is available. :modules:aggs-matrix-stats:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.018 secs. :modules:analysis-common:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:analysis-common:compileJava Task ':modules:analysis-common:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':modules:analysis-common:compileJava'. :modules:analysis-common:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\analysis-common Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\analysis-common\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Class dependency analysis for incremental compilation took 0.023 secs. Created jar classpath snapshot for incremental compilation in 0.003 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :modules:analysis-common:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.434 secs. :modules:analysis-common:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:analysis-common:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\analysis-common\src\main\resources', not found Skipping task ':modules:analysis-common:processResources' as it has no source files and no previous output files. :modules:analysis-common:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:analysis-common:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:analysis-common:classes Skipping task ':modules:analysis-common:classes' as it has no actions. :modules:analysis-common:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:analysis-common:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:analysis-common:jar Task ':modules:analysis-common:jar' is not up-to-date because: No history is available. :modules:analysis-common:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.03 secs. :modules:analysis-common:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:analysis-common:copyPluginPropertiesTemplate Task ':modules:analysis-common:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :modules:analysis-common:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :modules:analysis-common:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:analysis-common:pluginProperties Task ':modules:analysis-common:pluginProperties' is not up-to-date because: No history is available. :modules:analysis-common:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.015 secs. :modules:analysis-common:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:analysis-common:bundlePlugin Task ':modules:analysis-common:bundlePlugin' is not up-to-date because: No history is available. :modules:analysis-common:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.015 secs. :libs:grok:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:grok:compileJava Task ':libs:grok:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':libs:grok:compileJava'. :libs:grok:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\grok Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\grok\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.001 secs. Created jar classpath snapshot for incremental compilation in 0.07 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :libs:grok:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.426 secs. :libs:grok:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:grok:processResources Task ':libs:grok:processResources' is not up-to-date because: No history is available. :libs:grok:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.025 secs. :libs:grok:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:grok:classes Skipping task ':libs:grok:classes' as it has no actions. :libs:grok:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :libs:grok:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :libs:grok:jar Task ':libs:grok:jar' is not up-to-date because: No history is available. :libs:grok:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.029 secs. :modules:ingest-common:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:ingest-common:compileJava Task ':modules:ingest-common:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':modules:ingest-common:compileJava'. :modules:ingest-common:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\ingest-common Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\ingest-common\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. Class dependency analysis for incremental compilation took 0.025 secs. Created jar classpath snapshot for incremental compilation in 0.004 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :modules:ingest-common:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.195 secs. :modules:ingest-common:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:ingest-common:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\ingest-common\src\main\resources', not found Skipping task ':modules:ingest-common:processResources' as it has no source files and no previous output files. :modules:ingest-common:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:ingest-common:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:ingest-common:classes Skipping task ':modules:ingest-common:classes' as it has no actions. :modules:ingest-common:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:ingest-common:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:ingest-common:jar Task ':modules:ingest-common:jar' is not up-to-date because: No history is available. :modules:ingest-common:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.032 secs. :modules:ingest-common:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:ingest-common:copyPluginPropertiesTemplate Task ':modules:ingest-common:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :modules:ingest-common:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :modules:ingest-common:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:ingest-common:pluginProperties Task ':modules:ingest-common:pluginProperties' is not up-to-date because: No history is available. :modules:ingest-common:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.011 secs. :modules:ingest-common:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:ingest-common:bundlePlugin Task ':modules:ingest-common:bundlePlugin' is not up-to-date because: No history is available. :modules:ingest-common:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.178 secs. :modules:lang-expression:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-expressions/7.4.0-snapshot-1ed95c097b/lucene-expressions-7.4.0-snapshot-1ed95c097b.jar > Task :modules:lang-expression:compileJava Task ':modules:lang-expression:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':modules:lang-expression:compileJava'. :modules:lang-expression:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\lang-expression Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\lang-expression\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Class dependency analysis for incremental compilation took 0.013 secs. Created jar classpath snapshot for incremental compilation in 0.143 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :modules:lang-expression:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.72 secs. :modules:lang-expression:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-expression:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\lang-expression\src\main\resources', not found Skipping task ':modules:lang-expression:processResources' as it has no source files and no previous output files. :modules:lang-expression:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:lang-expression:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-expression:classes Skipping task ':modules:lang-expression:classes' as it has no actions. :modules:lang-expression:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:lang-expression:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-expression:jar Task ':modules:lang-expression:jar' is not up-to-date because: No history is available. :modules:lang-expression:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.021 secs. :modules:lang-expression:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-expression:copyPluginPropertiesTemplate Task ':modules:lang-expression:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :modules:lang-expression:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :modules:lang-expression:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-expression:pluginProperties Task ':modules:lang-expression:pluginProperties' is not up-to-date because: No history is available. :modules:lang-expression:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.016 secs. :modules:lang-expression:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-expression:bundlePlugin Task ':modules:lang-expression:bundlePlugin' is not up-to-date because: No history is available. :modules:lang-expression:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.051 secs. :modules:lang-mustache:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-mustache:compileJava Task ':modules:lang-mustache:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':modules:lang-mustache:compileJava'. :modules:lang-mustache:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\lang-mustache Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\lang-mustache\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\lang-mustache\src\main\java\org\elasticsearch\script\mustache\MustacheScriptEngine.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. Class dependency analysis for incremental compilation took 0.014 secs. Created jar classpath snapshot for incremental compilation in 0.02 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :modules:lang-mustache:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.052 secs. :modules:lang-mustache:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-mustache:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\lang-mustache\src\main\resources', not found Skipping task ':modules:lang-mustache:processResources' as it has no source files and no previous output files. :modules:lang-mustache:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:lang-mustache:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-mustache:classes Skipping task ':modules:lang-mustache:classes' as it has no actions. :modules:lang-mustache:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:lang-mustache:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-mustache:jar Task ':modules:lang-mustache:jar' is not up-to-date because: No history is available. :modules:lang-mustache:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.021 secs. :modules:lang-mustache:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-mustache:copyPluginPropertiesTemplate Task ':modules:lang-mustache:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :modules:lang-mustache:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :modules:lang-mustache:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-mustache:pluginProperties Task ':modules:lang-mustache:pluginProperties' is not up-to-date because: No history is available. :modules:lang-mustache:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.017 secs. :modules:lang-mustache:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-mustache:bundlePlugin Task ':modules:lang-mustache:bundlePlugin' is not up-to-date because: No history is available. :modules:lang-mustache:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.017 secs. :modules:lang-painless:spi:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-painless:spi:compileJava Task ':modules:lang-painless:spi:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':modules:lang-painless:spi:compileJava'. :modules:lang-painless:spi:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\lang-painless\spi Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\lang-painless\spi\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.004 secs. Created jar classpath snapshot for incremental compilation in 0.004 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :modules:lang-painless:spi:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.372 secs. :modules:lang-painless:spi:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-painless:spi:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\lang-painless\spi\src\main\resources', not found Skipping task ':modules:lang-painless:spi:processResources' as it has no source files and no previous output files. :modules:lang-painless:spi:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:lang-painless:spi:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-painless:spi:classes Skipping task ':modules:lang-painless:spi:classes' as it has no actions. :modules:lang-painless:spi:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:lang-painless:spi:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-painless:spi:jar Task ':modules:lang-painless:spi:jar' is not up-to-date because: No history is available. :modules:lang-painless:spi:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.013 secs. :modules:lang-painless:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-painless:compileJava Task ':modules:lang-painless:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':modules:lang-painless:compileJava'. :modules:lang-painless:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\lang-painless Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\lang-painless\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.09 secs. Created jar classpath snapshot for incremental compilation in 0.072 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :modules:lang-painless:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 4.869 secs. :modules:lang-painless:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-painless:processResources Task ':modules:lang-painless:processResources' is not up-to-date because: No history is available. :modules:lang-painless:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.024 secs. :modules:lang-painless:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-painless:classes Skipping task ':modules:lang-painless:classes' as it has no actions. :modules:lang-painless:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:lang-painless:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-painless:jar Task ':modules:lang-painless:jar' is not up-to-date because: No history is available. :modules:lang-painless:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.102 secs. :modules:lang-painless:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-painless:copyPluginPropertiesTemplate Task ':modules:lang-painless:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :modules:lang-painless:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:lang-painless:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-painless:pluginProperties Task ':modules:lang-painless:pluginProperties' is not up-to-date because: No history is available. :modules:lang-painless:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.012 secs. :modules:lang-painless:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:lang-painless:bundlePlugin Task ':modules:lang-painless:bundlePlugin' is not up-to-date because: No history is available. :modules:lang-painless:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.062 secs. :modules:mapper-extras:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:mapper-extras:compileJava Task ':modules:mapper-extras:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':modules:mapper-extras:compileJava'. :modules:mapper-extras:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\mapper-extras Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\mapper-extras\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.007 secs. Created jar classpath snapshot for incremental compilation in 0.003 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :modules:mapper-extras:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.636 secs. :modules:mapper-extras:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:mapper-extras:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\mapper-extras\src\main\resources', not found Skipping task ':modules:mapper-extras:processResources' as it has no source files and no previous output files. :modules:mapper-extras:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:mapper-extras:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:mapper-extras:classes Skipping task ':modules:mapper-extras:classes' as it has no actions. :modules:mapper-extras:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:mapper-extras:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:mapper-extras:jar Task ':modules:mapper-extras:jar' is not up-to-date because: No history is available. :modules:mapper-extras:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.02 secs. :modules:mapper-extras:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:mapper-extras:copyPluginPropertiesTemplate Task ':modules:mapper-extras:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :modules:mapper-extras:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :modules:mapper-extras:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:mapper-extras:pluginProperties Task ':modules:mapper-extras:pluginProperties' is not up-to-date because: No history is available. :modules:mapper-extras:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.011 secs. :modules:mapper-extras:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:mapper-extras:bundlePlugin Task ':modules:mapper-extras:bundlePlugin' is not up-to-date because: No history is available. :modules:mapper-extras:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.007 secs. :modules:parent-join:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:parent-join:compileJava Task ':modules:parent-join:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':modules:parent-join:compileJava'. :modules:parent-join:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\parent-join Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\parent-join\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.015 secs. Created jar classpath snapshot for incremental compilation in 0.003 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :modules:parent-join:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.186 secs. :modules:parent-join:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:parent-join:processResources Task ':modules:parent-join:processResources' is not up-to-date because: No history is available. :modules:parent-join:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.004 secs. :modules:parent-join:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:parent-join:classes Skipping task ':modules:parent-join:classes' as it has no actions. :modules:parent-join:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:parent-join:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:parent-join:jar Task ':modules:parent-join:jar' is not up-to-date because: No history is available. :modules:parent-join:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.027 secs. :modules:parent-join:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:parent-join:copyPluginPropertiesTemplate Task ':modules:parent-join:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :modules:parent-join:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:parent-join:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:parent-join:pluginProperties Task ':modules:parent-join:pluginProperties' is not up-to-date because: No history is available. :modules:parent-join:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.011 secs. :modules:parent-join:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:parent-join:bundlePlugin Task ':modules:parent-join:bundlePlugin' is not up-to-date because: No history is available. :modules:parent-join:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.011 secs. :modules:percolator:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:percolator:compileJava Task ':modules:percolator:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':modules:percolator:compileJava'. :modules:percolator:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\percolator Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\percolator\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Class dependency analysis for incremental compilation took 0.012 secs. Created jar classpath snapshot for incremental compilation in 0.005 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :modules:percolator:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.392 secs. :modules:percolator:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:percolator:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\percolator\src\main\resources', not found Skipping task ':modules:percolator:processResources' as it has no source files and no previous output files. :modules:percolator:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:percolator:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:percolator:classes Skipping task ':modules:percolator:classes' as it has no actions. :modules:percolator:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:percolator:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:percolator:jar Task ':modules:percolator:jar' is not up-to-date because: No history is available. :modules:percolator:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.021 secs. :modules:percolator:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:percolator:copyPluginPropertiesTemplate Task ':modules:percolator:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :modules:percolator:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :modules:percolator:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:percolator:pluginProperties Task ':modules:percolator:pluginProperties' is not up-to-date because: No history is available. :modules:percolator:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.011 secs. :modules:percolator:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:percolator:bundlePlugin Task ':modules:percolator:bundlePlugin' is not up-to-date because: No history is available. :modules:percolator:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.013 secs. :modules:rank-eval:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:rank-eval:compileJava Task ':modules:rank-eval:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':modules:rank-eval:compileJava'. :modules:rank-eval:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\rank-eval Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\rank-eval\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.014 secs. Created jar classpath snapshot for incremental compilation in 0.004 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :modules:rank-eval:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.296 secs. :modules:rank-eval:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:rank-eval:processResources Task ':modules:rank-eval:processResources' is not up-to-date because: No history is available. :modules:rank-eval:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.004 secs. :modules:rank-eval:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:rank-eval:classes Skipping task ':modules:rank-eval:classes' as it has no actions. :modules:rank-eval:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:rank-eval:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:rank-eval:jar Task ':modules:rank-eval:jar' is not up-to-date because: No history is available. :modules:rank-eval:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.022 secs. :modules:rank-eval:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:rank-eval:copyPluginPropertiesTemplate Task ':modules:rank-eval:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :modules:rank-eval:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:rank-eval:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:rank-eval:pluginProperties Task ':modules:rank-eval:pluginProperties' is not up-to-date because: No history is available. :modules:rank-eval:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.014 secs. :modules:rank-eval:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:rank-eval:bundlePlugin Task ':modules:rank-eval:bundlePlugin' is not up-to-date because: No history is available. :modules:rank-eval:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.015 secs. :modules:reindex:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:reindex:compileJava Task ':modules:reindex:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':modules:reindex:compileJava'. :modules:reindex:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\reindex Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\reindex\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Class dependency analysis for incremental compilation took 0.019 secs. Created jar classpath snapshot for incremental compilation in 0.004 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :modules:reindex:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.822 secs. :modules:reindex:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:reindex:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\reindex\src\main\resources', not found Skipping task ':modules:reindex:processResources' as it has no source files and no previous output files. :modules:reindex:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:reindex:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:reindex:classes Skipping task ':modules:reindex:classes' as it has no actions. :modules:reindex:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:reindex:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:reindex:jar Task ':modules:reindex:jar' is not up-to-date because: No history is available. :modules:reindex:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.027 secs. :modules:reindex:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:reindex:copyPluginPropertiesTemplate Task ':modules:reindex:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :modules:reindex:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :modules:reindex:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:reindex:pluginProperties Task ':modules:reindex:pluginProperties' is not up-to-date because: No history is available. :modules:reindex:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.01 secs. :modules:reindex:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:reindex:bundlePlugin Task ':modules:reindex:bundlePlugin' is not up-to-date because: No history is available. :modules:reindex:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.107 secs. :modules:repository-url:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:repository-url:compileJava Task ':modules:repository-url:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':modules:repository-url:compileJava'. :modules:repository-url:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\repository-url Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\repository-url\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.004 secs. Created jar classpath snapshot for incremental compilation in 0.003 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :modules:repository-url:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.501 secs. :modules:repository-url:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:repository-url:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\repository-url\src\main\resources', not found Skipping task ':modules:repository-url:processResources' as it has no source files and no previous output files. :modules:repository-url:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:repository-url:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:repository-url:classes Skipping task ':modules:repository-url:classes' as it has no actions. :modules:repository-url:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:repository-url:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:repository-url:jar Task ':modules:repository-url:jar' is not up-to-date because: No history is available. :modules:repository-url:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.014 secs. :modules:repository-url:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:repository-url:copyPluginPropertiesTemplate Task ':modules:repository-url:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :modules:repository-url:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.005 secs. :modules:repository-url:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:repository-url:pluginProperties Task ':modules:repository-url:pluginProperties' is not up-to-date because: No history is available. :modules:repository-url:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.027 secs. :modules:repository-url:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:repository-url:bundlePlugin Task ':modules:repository-url:bundlePlugin' is not up-to-date because: No history is available. :modules:repository-url:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.012 secs. :modules:transport-netty4:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:transport-netty4:compileJava Task ':modules:transport-netty4:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':modules:transport-netty4:compileJava'. :modules:transport-netty4:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\transport-netty4 Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\transport-netty4\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Class dependency analysis for incremental compilation took 0.017 secs. Created jar classpath snapshot for incremental compilation in 0.407 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :modules:transport-netty4:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 3.276 secs. :modules:transport-netty4:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:transport-netty4:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\modules\transport-netty4\src\main\resources', not found Skipping task ':modules:transport-netty4:processResources' as it has no source files and no previous output files. :modules:transport-netty4:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:transport-netty4:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:transport-netty4:classes Skipping task ':modules:transport-netty4:classes' as it has no actions. :modules:transport-netty4:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :modules:transport-netty4:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:transport-netty4:jar Task ':modules:transport-netty4:jar' is not up-to-date because: No history is available. :modules:transport-netty4:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.024 secs. :modules:transport-netty4:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:transport-netty4:copyPluginPropertiesTemplate Task ':modules:transport-netty4:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :modules:transport-netty4:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :modules:transport-netty4:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:transport-netty4:pluginProperties Task ':modules:transport-netty4:pluginProperties' is not up-to-date because: No history is available. :modules:transport-netty4:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.01 secs. :modules:transport-netty4:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :modules:transport-netty4:bundlePlugin Task ':modules:transport-netty4:bundlePlugin' is not up-to-date because: No history is available. :modules:transport-netty4:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.149 secs. :distribution:processOssOutputs (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:processOssOutputs Task ':distribution:processOssOutputs' is not up-to-date because: No history is available. :distribution:processOssOutputs (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.26 secs. :distribution:buildOssBin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:buildOssBin Skipping task ':distribution:buildOssBin' as it has no actions. :distribution:buildOssBin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :distribution:buildOssConfig (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:buildOssConfig Skipping task ':distribution:buildOssConfig' as it has no actions. :distribution:buildOssConfig (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :distribution:buildOssLog4jConfig (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:buildOssLog4jConfig Task ':distribution:buildOssLog4jConfig' is not up-to-date because: No history is available. :distribution:buildOssLog4jConfig (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.016 secs. :distribution:buildOssModules (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:buildOssModules Skipping task ':distribution:buildOssModules' as it has no actions. :distribution:buildOssModules (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :distribution:archives:createLogsDir (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:archives:createLogsDir Task ':distribution:archives:createLogsDir' is not up-to-date because: Task has not declared any outputs despite executing actions. :distribution:archives:createLogsDir (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.005 secs. :distribution:archives:createPluginsDir (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:archives:createPluginsDir Task ':distribution:archives:createPluginsDir' is not up-to-date because: Task has not declared any outputs despite executing actions. :distribution:archives:createPluginsDir (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :distribution:tools:launchers:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:tools:launchers:compileJava Task ':distribution:tools:launchers:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':distribution:tools:launchers:compileJava'. :distribution:tools:launchers:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\tools\launchers Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\tools\launchers\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.004 secs. Created jar classpath snapshot for incremental compilation in 0.0 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :distribution:tools:launchers:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.008 secs. :distribution:tools:launchers:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:tools:launchers:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\tools\launchers\src\main\resources', not found Skipping task ':distribution:tools:launchers:processResources' as it has no source files and no previous output files. :distribution:tools:launchers:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :distribution:tools:launchers:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:tools:launchers:classes Skipping task ':distribution:tools:launchers:classes' as it has no actions. :distribution:tools:launchers:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :distribution:tools:launchers:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:tools:launchers:jar Task ':distribution:tools:launchers:jar' is not up-to-date because: No history is available. :distribution:tools:launchers:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.015 secs. :distribution:tools:plugin-cli:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:tools:plugin-cli:compileJava Task ':distribution:tools:plugin-cli:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':distribution:tools:plugin-cli:compileJava'. :distribution:tools:plugin-cli:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\tools\plugin-cli Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\tools\plugin-cli\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\tools\plugin-cli\src\main\java\org\elasticsearch\plugins\InstallPluginCommand.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. Class dependency analysis for incremental compilation took 0.005 secs. Created jar classpath snapshot for incremental compilation in 0.004 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :distribution:tools:plugin-cli:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.83 secs. :distribution:tools:plugin-cli:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:tools:plugin-cli:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\distribution\tools\plugin-cli\src\main\resources', not found Skipping task ':distribution:tools:plugin-cli:processResources' as it has no source files and no previous output files. :distribution:tools:plugin-cli:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :distribution:tools:plugin-cli:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:tools:plugin-cli:classes Skipping task ':distribution:tools:plugin-cli:classes' as it has no actions. :distribution:tools:plugin-cli:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :distribution:tools:plugin-cli:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:tools:plugin-cli:jar Task ':distribution:tools:plugin-cli:jar' is not up-to-date because: No history is available. :distribution:tools:plugin-cli:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.015 secs. :distribution:archives:buildOssZip (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:archives:buildOssZip Task ':distribution:archives:buildOssZip' is not up-to-date because: No history is available. :distribution:archives:buildOssZip (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.769 secs. :docs:integTestCluster#extract (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#extract Task ':docs:integTestCluster#extract' is not up-to-date because: No history is available. :docs:integTestCluster#extract (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.659 secs. :plugins:analysis-icu:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:generateNotice Task ':plugins:analysis-icu:generateNotice' is not up-to-date because: No history is available. :plugins:analysis-icu:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.004 secs. :plugins:analysis-icu:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. Download https://repo.maven.apache.org/maven2/com/ibm/icu/icu4j/61.1/icu4j-61.1.jar Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-analyzers-icu/7.4.0-snapshot-1ed95c097b/lucene-analyzers-icu-7.4.0-snapshot-1ed95c097b.jar > Task :plugins:analysis-icu:compileJava Task ':plugins:analysis-icu:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:analysis-icu:compileJava'. :plugins:analysis-icu:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-icu Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-icu\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Class dependency analysis for incremental compilation took 0.01 secs. Created jar classpath snapshot for incremental compilation in 0.414 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:analysis-icu:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 3.317 secs. :plugins:analysis-icu:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-icu\src\main\resources', not found Skipping task ':plugins:analysis-icu:processResources' as it has no source files and no previous output files. :plugins:analysis-icu:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-icu:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:classes Skipping task ':plugins:analysis-icu:classes' as it has no actions. :plugins:analysis-icu:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-icu:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:jar Task ':plugins:analysis-icu:jar' is not up-to-date because: No history is available. :plugins:analysis-icu:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.018 secs. :plugins:analysis-icu:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:copyPluginPropertiesTemplate Task ':plugins:analysis-icu:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:analysis-icu:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:analysis-icu:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:pluginProperties Task ':plugins:analysis-icu:pluginProperties' is not up-to-date because: No history is available. :plugins:analysis-icu:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.011 secs. :plugins:analysis-icu:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:bundlePlugin Task ':plugins:analysis-icu:bundlePlugin' is not up-to-date because: No history is available. :plugins:analysis-icu:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.597 secs. :plugins:analysis-kuromoji:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-kuromoji:generateNotice Task ':plugins:analysis-kuromoji:generateNotice' is not up-to-date because: No history is available. :plugins:analysis-kuromoji:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.002 secs. :plugins:analysis-kuromoji:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-analyzers-kuromoji/7.4.0-snapshot-1ed95c097b/lucene-analyzers-kuromoji-7.4.0-snapshot-1ed95c097b.jar > Task :plugins:analysis-kuromoji:compileJava Task ':plugins:analysis-kuromoji:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:analysis-kuromoji:compileJava'. :plugins:analysis-kuromoji:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-kuromoji Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-kuromoji\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.006 secs. Created jar classpath snapshot for incremental compilation in 0.019 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:analysis-kuromoji:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 4.328 secs. :plugins:analysis-kuromoji:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-kuromoji:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-kuromoji\src\main\resources', not found Skipping task ':plugins:analysis-kuromoji:processResources' as it has no source files and no previous output files. :plugins:analysis-kuromoji:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-kuromoji:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-kuromoji:classes Skipping task ':plugins:analysis-kuromoji:classes' as it has no actions. :plugins:analysis-kuromoji:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-kuromoji:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-kuromoji:jar Task ':plugins:analysis-kuromoji:jar' is not up-to-date because: No history is available. :plugins:analysis-kuromoji:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.015 secs. :plugins:analysis-kuromoji:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-kuromoji:copyPluginPropertiesTemplate Task ':plugins:analysis-kuromoji:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:analysis-kuromoji:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:analysis-kuromoji:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-kuromoji:pluginProperties Task ':plugins:analysis-kuromoji:pluginProperties' is not up-to-date because: No history is available. :plugins:analysis-kuromoji:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.013 secs. :plugins:analysis-kuromoji:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-kuromoji:bundlePlugin Task ':plugins:analysis-kuromoji:bundlePlugin' is not up-to-date because: No history is available. :plugins:analysis-kuromoji:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.228 secs. :plugins:analysis-nori:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-nori:generateNotice Task ':plugins:analysis-nori:generateNotice' is not up-to-date because: No history is available. :plugins:analysis-nori:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.002 secs. :plugins:analysis-nori:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-analyzers-nori/7.4.0-snapshot-1ed95c097b/lucene-analyzers-nori-7.4.0-snapshot-1ed95c097b.jar > Task :plugins:analysis-nori:compileJava Task ':plugins:analysis-nori:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:analysis-nori:compileJava'. :plugins:analysis-nori:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-nori Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-nori\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.004 secs. Created jar classpath snapshot for incremental compilation in 0.016 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:analysis-nori:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 4.07 secs. :plugins:analysis-nori:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-nori:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-nori\src\main\resources', not found Skipping task ':plugins:analysis-nori:processResources' as it has no source files and no previous output files. :plugins:analysis-nori:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-nori:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-nori:classes Skipping task ':plugins:analysis-nori:classes' as it has no actions. :plugins:analysis-nori:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-nori:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-nori:jar Task ':plugins:analysis-nori:jar' is not up-to-date because: No history is available. :plugins:analysis-nori:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.013 secs. :plugins:analysis-nori:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-nori:copyPluginPropertiesTemplate Task ':plugins:analysis-nori:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:analysis-nori:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.002 secs. :plugins:analysis-nori:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-nori:pluginProperties Task ':plugins:analysis-nori:pluginProperties' is not up-to-date because: No history is available. :plugins:analysis-nori:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.011 secs. :plugins:analysis-nori:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-nori:bundlePlugin Task ':plugins:analysis-nori:bundlePlugin' is not up-to-date because: No history is available. :plugins:analysis-nori:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.346 secs. :plugins:analysis-phonetic:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-phonetic:generateNotice Task ':plugins:analysis-phonetic:generateNotice' is not up-to-date because: No history is available. :plugins:analysis-phonetic:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.002 secs. :plugins:analysis-phonetic:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-analyzers-phonetic/7.4.0-snapshot-1ed95c097b/lucene-analyzers-phonetic-7.4.0-snapshot-1ed95c097b.jar > Task :plugins:analysis-phonetic:compileJava Task ':plugins:analysis-phonetic:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:analysis-phonetic:compileJava'. :plugins:analysis-phonetic:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-phonetic Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-phonetic\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.005 secs. Created jar classpath snapshot for incremental compilation in 0.007 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:analysis-phonetic:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.925 secs. :plugins:analysis-phonetic:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-phonetic:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-phonetic\src\main\resources', not found Skipping task ':plugins:analysis-phonetic:processResources' as it has no source files and no previous output files. :plugins:analysis-phonetic:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-phonetic:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-phonetic:classes Skipping task ':plugins:analysis-phonetic:classes' as it has no actions. :plugins:analysis-phonetic:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-phonetic:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-phonetic:jar Task ':plugins:analysis-phonetic:jar' is not up-to-date because: No history is available. :plugins:analysis-phonetic:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.018 secs. :plugins:analysis-phonetic:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-phonetic:copyPluginPropertiesTemplate Task ':plugins:analysis-phonetic:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:analysis-phonetic:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:analysis-phonetic:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-phonetic:pluginProperties Task ':plugins:analysis-phonetic:pluginProperties' is not up-to-date because: No history is available. :plugins:analysis-phonetic:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.01 secs. :plugins:analysis-phonetic:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-phonetic:bundlePlugin Task ':plugins:analysis-phonetic:bundlePlugin' is not up-to-date because: No history is available. :plugins:analysis-phonetic:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.024 secs. :plugins:analysis-smartcn:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-smartcn:generateNotice Task ':plugins:analysis-smartcn:generateNotice' is not up-to-date because: No history is available. :plugins:analysis-smartcn:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.003 secs. :plugins:analysis-smartcn:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-analyzers-smartcn/7.4.0-snapshot-1ed95c097b/lucene-analyzers-smartcn-7.4.0-snapshot-1ed95c097b.jar > Task :plugins:analysis-smartcn:compileJava Task ':plugins:analysis-smartcn:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:analysis-smartcn:compileJava'. :plugins:analysis-smartcn:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-smartcn Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-smartcn\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.004 secs. Created jar classpath snapshot for incremental compilation in 0.01 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:analysis-smartcn:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 3.672 secs. :plugins:analysis-smartcn:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-smartcn:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-smartcn\src\main\resources', not found Skipping task ':plugins:analysis-smartcn:processResources' as it has no source files and no previous output files. :plugins:analysis-smartcn:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-smartcn:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-smartcn:classes Skipping task ':plugins:analysis-smartcn:classes' as it has no actions. :plugins:analysis-smartcn:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-smartcn:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-smartcn:jar Task ':plugins:analysis-smartcn:jar' is not up-to-date because: No history is available. :plugins:analysis-smartcn:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.012 secs. :plugins:analysis-smartcn:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-smartcn:copyPluginPropertiesTemplate Task ':plugins:analysis-smartcn:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:analysis-smartcn:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:analysis-smartcn:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-smartcn:pluginProperties Task ':plugins:analysis-smartcn:pluginProperties' is not up-to-date because: No history is available. :plugins:analysis-smartcn:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.015 secs. :plugins:analysis-smartcn:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-smartcn:bundlePlugin Task ':plugins:analysis-smartcn:bundlePlugin' is not up-to-date because: No history is available. :plugins:analysis-smartcn:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.194 secs. :plugins:analysis-stempel:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-stempel:generateNotice Task ':plugins:analysis-stempel:generateNotice' is not up-to-date because: No history is available. :plugins:analysis-stempel:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.002 secs. :plugins:analysis-stempel:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-analyzers-stempel/7.4.0-snapshot-1ed95c097b/lucene-analyzers-stempel-7.4.0-snapshot-1ed95c097b.jar > Task :plugins:analysis-stempel:compileJava Task ':plugins:analysis-stempel:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:analysis-stempel:compileJava'. :plugins:analysis-stempel:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-stempel Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-stempel\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.003 secs. Created jar classpath snapshot for incremental compilation in 0.01 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:analysis-stempel:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.37 secs. :plugins:analysis-stempel:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-stempel:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-stempel\src\main\resources', not found Skipping task ':plugins:analysis-stempel:processResources' as it has no source files and no previous output files. :plugins:analysis-stempel:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-stempel:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-stempel:classes Skipping task ':plugins:analysis-stempel:classes' as it has no actions. :plugins:analysis-stempel:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-stempel:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-stempel:jar Task ':plugins:analysis-stempel:jar' is not up-to-date because: No history is available. :plugins:analysis-stempel:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.013 secs. :plugins:analysis-stempel:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-stempel:copyPluginPropertiesTemplate Task ':plugins:analysis-stempel:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:analysis-stempel:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:analysis-stempel:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-stempel:pluginProperties Task ':plugins:analysis-stempel:pluginProperties' is not up-to-date because: No history is available. :plugins:analysis-stempel:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.018 secs. :plugins:analysis-stempel:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-stempel:bundlePlugin Task ':plugins:analysis-stempel:bundlePlugin' is not up-to-date because: No history is available. :plugins:analysis-stempel:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.087 secs. :plugins:analysis-ukrainian:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-ukrainian:generateNotice Task ':plugins:analysis-ukrainian:generateNotice' is not up-to-date because: No history is available. :plugins:analysis-ukrainian:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.006 secs. :plugins:analysis-ukrainian:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. Download http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/1ed95c097b/org/apache/lucene/lucene-analyzers-morfologik/7.4.0-snapshot-1ed95c097b/lucene-analyzers-morfologik-7.4.0-snapshot-1ed95c097b.jar > Task :plugins:analysis-ukrainian:compileJava Task ':plugins:analysis-ukrainian:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:analysis-ukrainian:compileJava'. :plugins:analysis-ukrainian:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-ukrainian Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-ukrainian\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.003 secs. Created jar classpath snapshot for incremental compilation in 0.02 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:analysis-ukrainian:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.602 secs. :plugins:analysis-ukrainian:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-ukrainian:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-ukrainian\src\main\resources', not found Skipping task ':plugins:analysis-ukrainian:processResources' as it has no source files and no previous output files. :plugins:analysis-ukrainian:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-ukrainian:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-ukrainian:classes Skipping task ':plugins:analysis-ukrainian:classes' as it has no actions. :plugins:analysis-ukrainian:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-ukrainian:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-ukrainian:jar Task ':plugins:analysis-ukrainian:jar' is not up-to-date because: No history is available. :plugins:analysis-ukrainian:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.012 secs. :plugins:analysis-ukrainian:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-ukrainian:copyPluginPropertiesTemplate Task ':plugins:analysis-ukrainian:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:analysis-ukrainian:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:analysis-ukrainian:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-ukrainian:pluginProperties Task ':plugins:analysis-ukrainian:pluginProperties' is not up-to-date because: No history is available. :plugins:analysis-ukrainian:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.01 secs. :plugins:analysis-ukrainian:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-ukrainian:bundlePlugin Task ':plugins:analysis-ukrainian:bundlePlugin' is not up-to-date because: No history is available. :plugins:analysis-ukrainian:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.185 secs. :plugins:discovery-azure-classic:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-azure-classic:generateNotice Task ':plugins:discovery-azure-classic:generateNotice' is not up-to-date because: No history is available. :plugins:discovery-azure-classic:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.006 secs. :plugins:discovery-azure-classic:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-azure-classic:compileJava Task ':plugins:discovery-azure-classic:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:discovery-azure-classic:compileJava'. :plugins:discovery-azure-classic:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-azure-classic Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-azure-classic\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.008 secs. Created jar classpath snapshot for incremental compilation in 0.902 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:discovery-azure-classic:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.992 secs. :plugins:discovery-azure-classic:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-azure-classic:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-azure-classic\src\main\resources', not found Skipping task ':plugins:discovery-azure-classic:processResources' as it has no source files and no previous output files. :plugins:discovery-azure-classic:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:discovery-azure-classic:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-azure-classic:classes Skipping task ':plugins:discovery-azure-classic:classes' as it has no actions. :plugins:discovery-azure-classic:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:discovery-azure-classic:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-azure-classic:jar Task ':plugins:discovery-azure-classic:jar' is not up-to-date because: No history is available. :plugins:discovery-azure-classic:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.016 secs. :plugins:discovery-azure-classic:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-azure-classic:copyPluginPropertiesTemplate Task ':plugins:discovery-azure-classic:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:discovery-azure-classic:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:discovery-azure-classic:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-azure-classic:pluginProperties Task ':plugins:discovery-azure-classic:pluginProperties' is not up-to-date because: No history is available. :plugins:discovery-azure-classic:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.01 secs. :plugins:discovery-azure-classic:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-azure-classic:bundlePlugin Task ':plugins:discovery-azure-classic:bundlePlugin' is not up-to-date because: No history is available. :plugins:discovery-azure-classic:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.303 secs. :plugins:discovery-ec2:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-ec2:generateNotice Task ':plugins:discovery-ec2:generateNotice' is not up-to-date because: No history is available. :plugins:discovery-ec2:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.003 secs. :plugins:discovery-ec2:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-ec2:compileJava Task ':plugins:discovery-ec2:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:discovery-ec2:compileJava'. :plugins:discovery-ec2:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-ec2 Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-ec2\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-ec2\src\main\java\org\elasticsearch\discovery\ec2\AwsEc2ServiceImpl.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. Class dependency analysis for incremental compilation took 0.006 secs. Created jar classpath snapshot for incremental compilation in 0.842 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:discovery-ec2:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 3.039 secs. :plugins:discovery-ec2:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-ec2:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-ec2\src\main\resources', not found Skipping task ':plugins:discovery-ec2:processResources' as it has no source files and no previous output files. :plugins:discovery-ec2:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:discovery-ec2:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-ec2:classes Skipping task ':plugins:discovery-ec2:classes' as it has no actions. :plugins:discovery-ec2:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:discovery-ec2:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-ec2:jar Task ':plugins:discovery-ec2:jar' is not up-to-date because: No history is available. :plugins:discovery-ec2:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.015 secs. :plugins:discovery-ec2:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-ec2:copyPluginPropertiesTemplate Task ':plugins:discovery-ec2:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:discovery-ec2:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:discovery-ec2:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-ec2:pluginProperties Task ':plugins:discovery-ec2:pluginProperties' is not up-to-date because: No history is available. :plugins:discovery-ec2:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.009 secs. :plugins:discovery-ec2:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-ec2:bundlePlugin Task ':plugins:discovery-ec2:bundlePlugin' is not up-to-date because: No history is available. :plugins:discovery-ec2:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.329 secs. :plugins:discovery-file:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-file:generateNotice Task ':plugins:discovery-file:generateNotice' is not up-to-date because: No history is available. :plugins:discovery-file:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:discovery-file:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-file:compileJava Task ':plugins:discovery-file:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:discovery-file:compileJava'. :plugins:discovery-file:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-file Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-file\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.002 secs. Created jar classpath snapshot for incremental compilation in 0.004 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:discovery-file:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.39 secs. :plugins:discovery-file:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-file:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-file\src\main\resources', not found Skipping task ':plugins:discovery-file:processResources' as it has no source files and no previous output files. :plugins:discovery-file:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:discovery-file:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-file:classes Skipping task ':plugins:discovery-file:classes' as it has no actions. :plugins:discovery-file:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:discovery-file:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-file:jar Task ':plugins:discovery-file:jar' is not up-to-date because: No history is available. :plugins:discovery-file:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.012 secs. :plugins:discovery-file:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-file:copyPluginPropertiesTemplate Task ':plugins:discovery-file:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:discovery-file:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:discovery-file:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-file:pluginProperties Task ':plugins:discovery-file:pluginProperties' is not up-to-date because: No history is available. :plugins:discovery-file:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.01 secs. :plugins:discovery-file:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-file:bundlePlugin Task ':plugins:discovery-file:bundlePlugin' is not up-to-date because: No history is available. :plugins:discovery-file:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.008 secs. :plugins:discovery-gce:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-gce:generateNotice Task ':plugins:discovery-gce:generateNotice' is not up-to-date because: No history is available. :plugins:discovery-gce:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.003 secs. :plugins:discovery-gce:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-gce:compileJava Task ':plugins:discovery-gce:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:discovery-gce:compileJava'. :plugins:discovery-gce:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-gce Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-gce\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.009 secs. Created jar classpath snapshot for incremental compilation in 0.293 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:discovery-gce:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.28 secs. :plugins:discovery-gce:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-gce:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\discovery-gce\src\main\resources', not found Skipping task ':plugins:discovery-gce:processResources' as it has no source files and no previous output files. :plugins:discovery-gce:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:discovery-gce:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-gce:classes Skipping task ':plugins:discovery-gce:classes' as it has no actions. :plugins:discovery-gce:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:discovery-gce:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-gce:jar Task ':plugins:discovery-gce:jar' is not up-to-date because: No history is available. :plugins:discovery-gce:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.017 secs. :plugins:discovery-gce:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-gce:copyPluginPropertiesTemplate Task ':plugins:discovery-gce:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:discovery-gce:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:discovery-gce:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-gce:pluginProperties Task ':plugins:discovery-gce:pluginProperties' is not up-to-date because: No history is available. :plugins:discovery-gce:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.014 secs. :plugins:discovery-gce:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:discovery-gce:bundlePlugin Task ':plugins:discovery-gce:bundlePlugin' is not up-to-date because: No history is available. :plugins:discovery-gce:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.157 secs. :plugins:ingest-attachment:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-attachment:generateNotice Task ':plugins:ingest-attachment:generateNotice' is not up-to-date because: No history is available. :plugins:ingest-attachment:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.008 secs. :plugins:ingest-attachment:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-attachment:compileJava Task ':plugins:ingest-attachment:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:ingest-attachment:compileJava'. :plugins:ingest-attachment:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\ingest-attachment Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\ingest-attachment\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\ingest-attachment\src\main\java\org\elasticsearch\ingest\attachment\AttachmentProcessor.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. Class dependency analysis for incremental compilation took 0.004 secs. Created jar classpath snapshot for incremental compilation in 4.386 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:ingest-attachment:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 7.575 secs. :plugins:ingest-attachment:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-attachment:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\ingest-attachment\src\main\resources', not found Skipping task ':plugins:ingest-attachment:processResources' as it has no source files and no previous output files. :plugins:ingest-attachment:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:ingest-attachment:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-attachment:classes Skipping task ':plugins:ingest-attachment:classes' as it has no actions. :plugins:ingest-attachment:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:ingest-attachment:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-attachment:jar Task ':plugins:ingest-attachment:jar' is not up-to-date because: No history is available. :plugins:ingest-attachment:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.012 secs. :plugins:ingest-attachment:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-attachment:copyPluginPropertiesTemplate Task ':plugins:ingest-attachment:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:ingest-attachment:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:ingest-attachment:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-attachment:pluginProperties Task ':plugins:ingest-attachment:pluginProperties' is not up-to-date because: No history is available. :plugins:ingest-attachment:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.011 secs. :plugins:ingest-attachment:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-attachment:bundlePlugin Task ':plugins:ingest-attachment:bundlePlugin' is not up-to-date because: No history is available. :plugins:ingest-attachment:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.295 secs. :plugins:ingest-geoip:copyDefaultGeoIp2DatabaseFiles (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-geoip:copyDefaultGeoIp2DatabaseFiles Task ':plugins:ingest-geoip:copyDefaultGeoIp2DatabaseFiles' is not up-to-date because: No history is available. :plugins:ingest-geoip:copyDefaultGeoIp2DatabaseFiles (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.394 secs. :plugins:ingest-geoip:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-geoip:generateNotice Task ':plugins:ingest-geoip:generateNotice' is not up-to-date because: No history is available. :plugins:ingest-geoip:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.002 secs. :plugins:ingest-geoip:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-geoip:compileJava Task ':plugins:ingest-geoip:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:ingest-geoip:compileJava'. :plugins:ingest-geoip:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\ingest-geoip Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\ingest-geoip\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.005 secs. Created jar classpath snapshot for incremental compilation in 0.215 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:ingest-geoip:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.874 secs. :plugins:ingest-geoip:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-geoip:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\ingest-geoip\src\main\resources', not found Skipping task ':plugins:ingest-geoip:processResources' as it has no source files and no previous output files. :plugins:ingest-geoip:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:ingest-geoip:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-geoip:classes Skipping task ':plugins:ingest-geoip:classes' as it has no actions. :plugins:ingest-geoip:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:ingest-geoip:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-geoip:jar Task ':plugins:ingest-geoip:jar' is not up-to-date because: No history is available. :plugins:ingest-geoip:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.014 secs. :plugins:ingest-geoip:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-geoip:copyPluginPropertiesTemplate Task ':plugins:ingest-geoip:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:ingest-geoip:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:ingest-geoip:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-geoip:pluginProperties Task ':plugins:ingest-geoip:pluginProperties' is not up-to-date because: No history is available. :plugins:ingest-geoip:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.009 secs. :plugins:ingest-geoip:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-geoip:bundlePlugin Task ':plugins:ingest-geoip:bundlePlugin' is not up-to-date because: No history is available. :plugins:ingest-geoip:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 6.063 secs. :plugins:ingest-user-agent:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-user-agent:generateNotice Task ':plugins:ingest-user-agent:generateNotice' is not up-to-date because: No history is available. :plugins:ingest-user-agent:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:ingest-user-agent:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-user-agent:compileJava Task ':plugins:ingest-user-agent:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:ingest-user-agent:compileJava'. :plugins:ingest-user-agent:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\ingest-user-agent Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\ingest-user-agent\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.006 secs. Created jar classpath snapshot for incremental compilation in 0.004 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:ingest-user-agent:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.495 secs. :plugins:ingest-user-agent:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-user-agent:processResources Task ':plugins:ingest-user-agent:processResources' is not up-to-date because: No history is available. :plugins:ingest-user-agent:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.004 secs. :plugins:ingest-user-agent:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-user-agent:classes Skipping task ':plugins:ingest-user-agent:classes' as it has no actions. :plugins:ingest-user-agent:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:ingest-user-agent:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-user-agent:jar Task ':plugins:ingest-user-agent:jar' is not up-to-date because: No history is available. :plugins:ingest-user-agent:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.021 secs. :plugins:ingest-user-agent:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-user-agent:copyPluginPropertiesTemplate Task ':plugins:ingest-user-agent:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:ingest-user-agent:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:ingest-user-agent:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-user-agent:pluginProperties Task ':plugins:ingest-user-agent:pluginProperties' is not up-to-date because: No history is available. :plugins:ingest-user-agent:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.015 secs. :plugins:ingest-user-agent:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:ingest-user-agent:bundlePlugin Task ':plugins:ingest-user-agent:bundlePlugin' is not up-to-date because: No history is available. :plugins:ingest-user-agent:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.013 secs. :plugins:mapper-murmur3:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:mapper-murmur3:generateNotice Task ':plugins:mapper-murmur3:generateNotice' is not up-to-date because: No history is available. :plugins:mapper-murmur3:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.002 secs. :plugins:mapper-murmur3:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:mapper-murmur3:compileJava Task ':plugins:mapper-murmur3:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:mapper-murmur3:compileJava'. :plugins:mapper-murmur3:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\mapper-murmur3 Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\mapper-murmur3\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.004 secs. Created jar classpath snapshot for incremental compilation in 0.003 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:mapper-murmur3:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.281 secs. :plugins:mapper-murmur3:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:mapper-murmur3:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\mapper-murmur3\src\main\resources', not found Skipping task ':plugins:mapper-murmur3:processResources' as it has no source files and no previous output files. :plugins:mapper-murmur3:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:mapper-murmur3:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:mapper-murmur3:classes Skipping task ':plugins:mapper-murmur3:classes' as it has no actions. :plugins:mapper-murmur3:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:mapper-murmur3:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:mapper-murmur3:jar Task ':plugins:mapper-murmur3:jar' is not up-to-date because: No history is available. :plugins:mapper-murmur3:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.015 secs. :plugins:mapper-murmur3:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:mapper-murmur3:copyPluginPropertiesTemplate Task ':plugins:mapper-murmur3:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:mapper-murmur3:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:mapper-murmur3:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:mapper-murmur3:pluginProperties Task ':plugins:mapper-murmur3:pluginProperties' is not up-to-date because: No history is available. :plugins:mapper-murmur3:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.01 secs. :plugins:mapper-murmur3:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:mapper-murmur3:bundlePlugin Task ':plugins:mapper-murmur3:bundlePlugin' is not up-to-date because: No history is available. :plugins:mapper-murmur3:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.008 secs. :plugins:mapper-size:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:mapper-size:generateNotice Task ':plugins:mapper-size:generateNotice' is not up-to-date because: No history is available. :plugins:mapper-size:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:mapper-size:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:mapper-size:compileJava Task ':plugins:mapper-size:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:mapper-size:compileJava'. :plugins:mapper-size:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\mapper-size Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\mapper-size\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.004 secs. Created jar classpath snapshot for incremental compilation in 0.005 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:mapper-size:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.298 secs. :plugins:mapper-size:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:mapper-size:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\mapper-size\src\main\resources', not found Skipping task ':plugins:mapper-size:processResources' as it has no source files and no previous output files. :plugins:mapper-size:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:mapper-size:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:mapper-size:classes Skipping task ':plugins:mapper-size:classes' as it has no actions. :plugins:mapper-size:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:mapper-size:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:mapper-size:jar Task ':plugins:mapper-size:jar' is not up-to-date because: No history is available. :plugins:mapper-size:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.018 secs. :plugins:mapper-size:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:mapper-size:copyPluginPropertiesTemplate Task ':plugins:mapper-size:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:mapper-size:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:mapper-size:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:mapper-size:pluginProperties Task ':plugins:mapper-size:pluginProperties' is not up-to-date because: No history is available. :plugins:mapper-size:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.013 secs. :plugins:mapper-size:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:mapper-size:bundlePlugin Task ':plugins:mapper-size:bundlePlugin' is not up-to-date because: No history is available. :plugins:mapper-size:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.01 secs. :plugins:store-smb:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:store-smb:generateNotice Task ':plugins:store-smb:generateNotice' is not up-to-date because: No history is available. :plugins:store-smb:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:store-smb:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:store-smb:compileJava Task ':plugins:store-smb:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:store-smb:compileJava'. :plugins:store-smb:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\store-smb Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\store-smb\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.005 secs. Created jar classpath snapshot for incremental compilation in 0.003 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:store-smb:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.375 secs. :plugins:store-smb:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:store-smb:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\store-smb\src\main\resources', not found Skipping task ':plugins:store-smb:processResources' as it has no source files and no previous output files. :plugins:store-smb:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:store-smb:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:store-smb:classes Skipping task ':plugins:store-smb:classes' as it has no actions. :plugins:store-smb:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:store-smb:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:store-smb:jar Task ':plugins:store-smb:jar' is not up-to-date because: No history is available. :plugins:store-smb:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.014 secs. :plugins:store-smb:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:store-smb:copyPluginPropertiesTemplate Task ':plugins:store-smb:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:store-smb:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:store-smb:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:store-smb:pluginProperties Task ':plugins:store-smb:pluginProperties' is not up-to-date because: No history is available. :plugins:store-smb:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.01 secs. :plugins:store-smb:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:store-smb:bundlePlugin Task ':plugins:store-smb:bundlePlugin' is not up-to-date because: No history is available. :plugins:store-smb:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.01 secs. :plugins:transport-nio:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:transport-nio:generateNotice Task ':plugins:transport-nio:generateNotice' is not up-to-date because: No history is available. :plugins:transport-nio:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.006 secs. :plugins:transport-nio:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:transport-nio:compileJava Task ':plugins:transport-nio:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':plugins:transport-nio:compileJava'. :plugins:transport-nio:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\transport-nio Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\transport-nio\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.004 secs. Created jar classpath snapshot for incremental compilation in 0.004 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :plugins:transport-nio:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.525 secs. :plugins:transport-nio:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:transport-nio:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\transport-nio\src\main\resources', not found Skipping task ':plugins:transport-nio:processResources' as it has no source files and no previous output files. :plugins:transport-nio:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:transport-nio:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:transport-nio:classes Skipping task ':plugins:transport-nio:classes' as it has no actions. :plugins:transport-nio:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:transport-nio:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:transport-nio:jar Task ':plugins:transport-nio:jar' is not up-to-date because: No history is available. :plugins:transport-nio:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.02 secs. :plugins:transport-nio:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:transport-nio:copyPluginPropertiesTemplate Task ':plugins:transport-nio:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:transport-nio:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:transport-nio:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:transport-nio:pluginProperties Task ':plugins:transport-nio:pluginProperties' is not up-to-date because: No history is available. :plugins:transport-nio:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.015 secs. :plugins:transport-nio:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:transport-nio:bundlePlugin Task ':plugins:transport-nio:bundlePlugin' is not up-to-date because: No history is available. :plugins:transport-nio:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.017 secs. :docs:integTestCluster#configure (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#configure Task ':docs:integTestCluster#configure' is not up-to-date because: Task has not declared any outputs despite executing actions. Configuring C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\elasticsearch-7.0.0-alpha1-SNAPSHOT\config\elasticsearch.yml :docs:integTestCluster#configure (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.002 secs. :docs:integTestCluster#copyPlugins (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#copyPlugins Task ':docs:integTestCluster#copyPlugins' is not up-to-date because: No history is available. :docs:integTestCluster#copyPlugins (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.702 secs. :docs:integTestCluster#installAnalysisIcuPlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installAnalysisIcuPlugin Task ':docs:integTestCluster#installAnalysisIcuPlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/analysis-icu-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/analysis-icu-7.0.0-alpha1-SNAPSHOT.zip [ ] 1%?? [> ] 2%?? [> ] 3%?? [=> ] 4%?? [=> ] 5%?? [==> ] 6%?? [==> ] 7%?? [===> ] 8%?? [===> ] 9%?? [====> ] 10%?? [====> ] 11%?? [=====> ] 12%?? [=====> ] 13%?? [======> ] 14%?? [======> ] 15%?? [=======> ] 16%?? [=======> ] 17%?? [========> ] 18%?? [========> ] 19%?? [=========> ] 20%?? [=========> ] 21%?? [==========> ] 22%?? [==========> ] 23%?? [===========> ] 24%?? [===========> ] 25%?? [============> ] 26%?? [============> ] 27%?? [=============> ] 28%?? [=============> ] 29%?? [==============> ] 30%?? [==============> ] 31%?? [===============> ] 32%?? [===============> ] 33%?? [================> ] 34%?? [================> ] 35%?? [=================> ] 36%?? [=================> ] 37%?? [==================> ] 38%?? [==================> ] 39%?? [===================> ] 40%?? [===================> ] 41%?? [====================> ] 42%?? [====================> ] 43%?? [=====================> ] 44%?? [=====================> ] 45%?? [======================> ] 46%?? [======================> ] 47%?? [=======================> ] 48%?? [=======================> ] 49%?? [========================> ] 50%?? [========================> ] 51%?? [=========================> ] 52%?? [=========================> ] 53%?? [==========================> ] 54%?? [==========================> ] 55%?? [===========================> ] 56%?? [===========================> ] 57%?? [============================> ] 58%?? [============================> ] 59%?? [=============================> ] 60%?? [=============================> ] 61%?? [==============================> ] 62%?? [==============================> ] 63%?? [===============================> ] 64%?? [===============================> ] 65%?? [================================> ] 66%?? [================================> ] 67%?? [=================================> ] 68%?? [=================================> ] 69%?? [==================================> ] 70%?? [==================================> ] 71%?? [===================================> ] 72%?? [===================================> ] 73%?? [====================================> ] 74%?? [====================================> ] 75%?? [=====================================> ] 76%?? [=====================================> ] 77%?? [======================================> ] 78%?? [======================================> ] 79%?? [=======================================> ] 80%?? [=======================================> ] 81%?? [========================================> ] 82%?? [========================================> ] 83%?? [=========================================> ] 84%?? [=========================================> ] 85%?? [==========================================> ] 86%?? [==========================================> ] 87%?? [===========================================> ] 88%?? [===========================================> ] 89%?? [============================================> ] 90%?? [============================================> ] 91%?? [=============================================> ] 92%?? [=============================================> ] 93%?? [==============================================> ] 94%?? [==============================================> ] 95%?? [===============================================> ] 96%?? [===============================================> ] 97%?? [================================================>] 98%?? [================================================>] 99%?? [=================================================] 100%?? -> Installed analysis-icu :docs:integTestCluster#installAnalysisIcuPlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.979 secs. :docs:integTestCluster#installAnalysisKuromojiPlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installAnalysisKuromojiPlugin Task ':docs:integTestCluster#installAnalysisKuromojiPlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/analysis-kuromoji-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/analysis-kuromoji-7.0.0-alpha1-SNAPSHOT.zip [ ] 1%?? [> ] 2%?? [> ] 3%?? [=> ] 4%?? [=> ] 5%?? [==> ] 6%?? [==> ] 7%?? [===> ] 8%?? [===> ] 9%?? [====> ] 10%?? [====> ] 11%?? [=====> ] 12%?? [=====> ] 13%?? [======> ] 14%?? [======> ] 15%?? [=======> ] 16%?? [=======> ] 17%?? [========> ] 18%?? [========> ] 19%?? [=========> ] 20%?? [=========> ] 21%?? [==========> ] 22%?? [==========> ] 23%?? [===========> ] 24%?? [===========> ] 25%?? [============> ] 26%?? [============> ] 27%?? [=============> ] 28%?? [=============> ] 29%?? [==============> ] 30%?? [==============> ] 31%?? [===============> ] 32%?? [===============> ] 33%?? [================> ] 34%?? [================> ] 35%?? [=================> ] 36%?? [=================> ] 37%?? [==================> ] 38%?? [==================> ] 39%?? [===================> ] 40%?? [===================> ] 41%?? [====================> ] 42%?? [====================> ] 43%?? [=====================> ] 44%?? [=====================> ] 45%?? [======================> ] 46%?? [======================> ] 47%?? [=======================> ] 48%?? [=======================> ] 49%?? [========================> ] 50%?? [========================> ] 51%?? [=========================> ] 52%?? [=========================> ] 53%?? [==========================> ] 54%?? [==========================> ] 55%?? [===========================> ] 56%?? [===========================> ] 57%?? [============================> ] 58%?? [============================> ] 59%?? [=============================> ] 60%?? [=============================> ] 61%?? [==============================> ] 62%?? [==============================> ] 63%?? [===============================> ] 64%?? [===============================> ] 65%?? [================================> ] 66%?? [================================> ] 67%?? [=================================> ] 68%?? [=================================> ] 69%?? [==================================> ] 70%?? [==================================> ] 71%?? [===================================> ] 72%?? [===================================> ] 73%?? [====================================> ] 74%?? [====================================> ] 75%?? [=====================================> ] 76%?? [=====================================> ] 77%?? [======================================> ] 78%?? [======================================> ] 79%?? [=======================================> ] 80%?? [=======================================> ] 81%?? [========================================> ] 82%?? [========================================> ] 83%?? [=========================================> ] 84%?? [=========================================> ] 85%?? [==========================================> ] 86%?? [==========================================> ] 87%?? [===========================================> ] 88%?? [===========================================> ] 89%?? [============================================> ] 90%?? [============================================> ] 91%?? [=============================================> ] 92%?? [=============================================> ] 93%?? [==============================================> ] 94%?? [==============================================> ] 95%?? [===============================================> ] 96%?? [===============================================> ] 97%?? [================================================>] 98%?? [================================================>] 99%?? [=================================================] 100%?? -> Installed analysis-kuromoji :docs:integTestCluster#installAnalysisKuromojiPlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.91 secs. :docs:integTestCluster#installAnalysisNoriPlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installAnalysisNoriPlugin Task ':docs:integTestCluster#installAnalysisNoriPlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/analysis-nori-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/analysis-nori-7.0.0-alpha1-SNAPSHOT.zip [ ] 1%?? [> ] 2%?? [> ] 3%?? [=> ] 4%?? [=> ] 5%?? [==> ] 6%?? [==> ] 7%?? [===> ] 8%?? [===> ] 9%?? [====> ] 10%?? [====> ] 11%?? [=====> ] 12%?? [=====> ] 13%?? [======> ] 14%?? [======> ] 15%?? [=======> ] 16%?? [=======> ] 17%?? [========> ] 18%?? [========> ] 19%?? [=========> ] 20%?? [=========> ] 21%?? [==========> ] 22%?? [==========> ] 23%?? [===========> ] 24%?? [===========> ] 25%?? [============> ] 26%?? [============> ] 27%?? [=============> ] 28%?? [=============> ] 29%?? [==============> ] 30%?? [==============> ] 31%?? [===============> ] 32%?? [===============> ] 33%?? [================> ] 34%?? [================> ] 35%?? [=================> ] 36%?? [=================> ] 37%?? [==================> ] 38%?? [==================> ] 39%?? [===================> ] 40%?? [===================> ] 41%?? [====================> ] 42%?? [====================> ] 43%?? [=====================> ] 44%?? [=====================> ] 45%?? [======================> ] 46%?? [======================> ] 47%?? [=======================> ] 48%?? [=======================> ] 49%?? [========================> ] 50%?? [========================> ] 51%?? [=========================> ] 52%?? [=========================> ] 53%?? [==========================> ] 54%?? [==========================> ] 55%?? [===========================> ] 56%?? [===========================> ] 57%?? [============================> ] 58%?? [============================> ] 59%?? [=============================> ] 60%?? [=============================> ] 61%?? [==============================> ] 62%?? [==============================> ] 63%?? [===============================> ] 64%?? [===============================> ] 65%?? [================================> ] 66%?? [================================> ] 67%?? [=================================> ] 68%?? [=================================> ] 69%?? [==================================> ] 70%?? [==================================> ] 71%?? [===================================> ] 72%?? [===================================> ] 73%?? [====================================> ] 74%?? [====================================> ] 75%?? [=====================================> ] 76%?? [=====================================> ] 77%?? [======================================> ] 78%?? [======================================> ] 79%?? [=======================================> ] 80%?? [=======================================> ] 81%?? [========================================> ] 82%?? [========================================> ] 83%?? [=========================================> ] 84%?? [=========================================> ] 85%?? [==========================================> ] 86%?? [==========================================> ] 87%?? [===========================================> ] 88%?? [===========================================> ] 89%?? [============================================> ] 90%?? [============================================> ] 91%?? [=============================================> ] 92%?? [=============================================> ] 93%?? [==============================================> ] 94%?? [==============================================> ] 95%?? [===============================================> ] 96%?? [===============================================> ] 97%?? [================================================>] 98%?? [================================================>] 99%?? [=================================================] 100%?? -> Installed analysis-nori :docs:integTestCluster#installAnalysisNoriPlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.916 secs. :docs:integTestCluster#installAnalysisPhoneticPlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installAnalysisPhoneticPlugin Task ':docs:integTestCluster#installAnalysisPhoneticPlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/analysis-phonetic-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/analysis-phonetic-7.0.0-alpha1-SNAPSHOT.zip [> ] 2%?? [=> ] 5%?? [===> ] 8%?? [====> ] 11%?? [======> ] 14%?? [=======> ] 16%?? [========> ] 19%?? [==========> ] 22%?? [===========> ] 25%?? [=============> ] 28%?? [==============> ] 31%?? [===============> ] 33%?? [=================> ] 36%?? [==================> ] 39%?? [====================> ] 42%?? [=====================> ] 45%?? [======================> ] 47%?? [========================> ] 50%?? [=========================> ] 53%?? [===========================> ] 56%?? [============================> ] 59%?? [==============================> ] 62%?? [===============================> ] 64%?? [================================> ] 67%?? [==================================> ] 70%?? [===================================> ] 73%?? [=====================================> ] 76%?? [======================================> ] 78%?? [=======================================> ] 81%?? [=========================================> ] 84%?? [==========================================> ] 87%?? [============================================> ] 90%?? [=============================================> ] 93%?? [==============================================> ] 95%?? [================================================>] 98%?? [================================================>] 99%?? [=================================================] 100%?? -> Installed analysis-phonetic :docs:integTestCluster#installAnalysisPhoneticPlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.86 secs. :docs:integTestCluster#installAnalysisSmartcnPlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installAnalysisSmartcnPlugin Task ':docs:integTestCluster#installAnalysisSmartcnPlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/analysis-smartcn-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/analysis-smartcn-7.0.0-alpha1-SNAPSHOT.zip [ ] 1%?? [> ] 2%?? [> ] 3%?? [=> ] 4%?? [=> ] 5%?? [==> ] 6%?? [==> ] 7%?? [===> ] 8%?? [===> ] 9%?? [====> ] 10%?? [====> ] 11%?? [=====> ] 12%?? [=====> ] 13%?? [======> ] 14%?? [======> ] 15%?? [=======> ] 16%?? [=======> ] 17%?? [========> ] 18%?? [========> ] 19%?? [=========> ] 20%?? [=========> ] 21%?? [==========> ] 22%?? [==========> ] 23%?? [===========> ] 24%?? [===========> ] 25%?? [============> ] 26%?? [============> ] 27%?? [=============> ] 28%?? [=============> ] 29%?? [==============> ] 30%?? [==============> ] 31%?? [===============> ] 32%?? [===============> ] 33%?? [================> ] 34%?? [================> ] 35%?? [=================> ] 36%?? [=================> ] 37%?? [==================> ] 38%?? [==================> ] 39%?? [===================> ] 40%?? [===================> ] 41%?? [====================> ] 42%?? [====================> ] 43%?? [=====================> ] 44%?? [=====================> ] 45%?? [======================> ] 46%?? [======================> ] 47%?? [=======================> ] 48%?? [=======================> ] 49%?? [========================> ] 50%?? [========================> ] 51%?? [=========================> ] 52%?? [=========================> ] 53%?? [==========================> ] 54%?? [==========================> ] 55%?? [===========================> ] 56%?? [===========================> ] 57%?? [============================> ] 58%?? [============================> ] 59%?? [=============================> ] 60%?? [=============================> ] 61%?? [==============================> ] 62%?? [==============================> ] 63%?? [===============================> ] 64%?? [===============================> ] 65%?? [================================> ] 66%?? [================================> ] 67%?? [=================================> ] 68%?? [=================================> ] 69%?? [==================================> ] 70%?? [==================================> ] 71%?? [===================================> ] 72%?? [===================================> ] 73%?? [====================================> ] 74%?? [====================================> ] 75%?? [=====================================> ] 76%?? [=====================================> ] 77%?? [======================================> ] 78%?? [======================================> ] 79%?? [=======================================> ] 80%?? [=======================================> ] 81%?? [========================================> ] 82%?? [========================================> ] 83%?? [=========================================> ] 84%?? [=========================================> ] 85%?? [==========================================> ] 86%?? [==========================================> ] 87%?? [===========================================> ] 88%?? [===========================================> ] 89%?? [============================================> ] 90%?? [============================================> ] 91%?? [=============================================> ] 92%?? [=============================================> ] 93%?? [==============================================> ] 94%?? [==============================================> ] 95%?? [===============================================> ] 96%?? [===============================================> ] 97%?? [================================================>] 98%?? [================================================>] 99%?? [=================================================] 100%?? -> Installed analysis-smartcn :docs:integTestCluster#installAnalysisSmartcnPlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.878 secs. :docs:integTestCluster#installAnalysisStempelPlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installAnalysisStempelPlugin Task ':docs:integTestCluster#installAnalysisStempelPlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/analysis-stempel-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/analysis-stempel-7.0.0-alpha1-SNAPSHOT.zip [ ] 1%?? [> ] 3%?? [=> ] 4%?? [==> ] 6%?? [==> ] 7%?? [===> ] 9%?? [====> ] 10%?? [=====> ] 12%?? [=====> ] 13%?? [======> ] 15%?? [=======> ] 16%?? [========> ] 18%?? [========> ] 19%?? [=========> ] 21%?? [==========> ] 22%?? [===========> ] 24%?? [===========> ] 25%?? [============> ] 27%?? [=============> ] 28%?? [==============> ] 30%?? [==============> ] 31%?? [===============> ] 33%?? [================> ] 35%?? [=================> ] 36%?? [==================> ] 38%?? [==================> ] 39%?? [===================> ] 41%?? [====================> ] 42%?? [=====================> ] 44%?? [=====================> ] 45%?? [======================> ] 47%?? [=======================> ] 48%?? [========================> ] 50%?? [========================> ] 51%?? [=========================> ] 53%?? [==========================> ] 54%?? [===========================> ] 56%?? [===========================> ] 57%?? [============================> ] 59%?? [=============================> ] 60%?? [==============================> ] 62%?? [==============================> ] 63%?? [===============================> ] 65%?? [================================> ] 67%?? [=================================> ] 68%?? [==================================> ] 70%?? [==================================> ] 71%?? [===================================> ] 73%?? [====================================> ] 74%?? [=====================================> ] 76%?? [=====================================> ] 77%?? [======================================> ] 79%?? [=======================================> ] 80%?? [========================================> ] 82%?? [========================================> ] 83%?? [=========================================> ] 85%?? [==========================================> ] 86%?? [===========================================> ] 88%?? [===========================================> ] 89%?? [============================================> ] 91%?? [=============================================> ] 92%?? [==============================================> ] 94%?? [==============================================> ] 95%?? [===============================================> ] 97%?? [================================================>] 99%?? [=================================================] 100%?? -> Installed analysis-stempel :docs:integTestCluster#installAnalysisStempelPlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.892 secs. :docs:integTestCluster#installAnalysisUkrainianPlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installAnalysisUkrainianPlugin Task ':docs:integTestCluster#installAnalysisUkrainianPlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/analysis-ukrainian-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/analysis-ukrainian-7.0.0-alpha1-SNAPSHOT.zip [ ] 1%?? [> ] 2%?? [> ] 3%?? [=> ] 4%?? [=> ] 5%?? [==> ] 6%?? [==> ] 7%?? [===> ] 8%?? [===> ] 9%?? [====> ] 10%?? [====> ] 11%?? [=====> ] 12%?? [=====> ] 13%?? [======> ] 14%?? [======> ] 15%?? [=======> ] 16%?? [=======> ] 17%?? [========> ] 18%?? [========> ] 19%?? [=========> ] 20%?? [=========> ] 21%?? [==========> ] 22%?? [==========> ] 23%?? [===========> ] 24%?? [===========> ] 25%?? [============> ] 26%?? [============> ] 27%?? [=============> ] 28%?? [=============> ] 29%?? [==============> ] 30%?? [==============> ] 31%?? [===============> ] 32%?? [===============> ] 33%?? [================> ] 34%?? [================> ] 35%?? [=================> ] 36%?? [=================> ] 37%?? [==================> ] 38%?? [==================> ] 39%?? [===================> ] 40%?? [===================> ] 41%?? [====================> ] 42%?? [====================> ] 43%?? [=====================> ] 44%?? [=====================> ] 45%?? [======================> ] 46%?? [======================> ] 47%?? [=======================> ] 48%?? [=======================> ] 49%?? [========================> ] 50%?? [========================> ] 51%?? [=========================> ] 52%?? [=========================> ] 53%?? [==========================> ] 54%?? [==========================> ] 55%?? [===========================> ] 56%?? [===========================> ] 57%?? [============================> ] 58%?? [============================> ] 59%?? [=============================> ] 60%?? [=============================> ] 61%?? [==============================> ] 62%?? [==============================> ] 63%?? [===============================> ] 64%?? [===============================> ] 65%?? [================================> ] 66%?? [================================> ] 67%?? [=================================> ] 68%?? [=================================> ] 69%?? [==================================> ] 70%?? [==================================> ] 71%?? [===================================> ] 72%?? [===================================> ] 73%?? [====================================> ] 74%?? [====================================> ] 75%?? [=====================================> ] 76%?? [=====================================> ] 77%?? [======================================> ] 78%?? [======================================> ] 79%?? [=======================================> ] 80%?? [=======================================> ] 81%?? [========================================> ] 82%?? [========================================> ] 83%?? [=========================================> ] 84%?? [=========================================> ] 85%?? [==========================================> ] 86%?? [==========================================> ] 87%?? [===========================================> ] 88%?? [===========================================> ] 89%?? [============================================> ] 90%?? [============================================> ] 91%?? [=============================================> ] 92%?? [=============================================> ] 93%?? [==============================================> ] 94%?? [==============================================> ] 95%?? [===============================================> ] 96%?? [===============================================> ] 97%?? [================================================>] 98%?? [================================================>] 99%?? [=================================================] 100%?? -> Installed analysis-ukrainian :docs:integTestCluster#installAnalysisUkrainianPlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.029 secs. :docs:integTestCluster#installDiscoveryAzureClassicPlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installDiscoveryAzureClassicPlugin Task ':docs:integTestCluster#installDiscoveryAzureClassicPlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/discovery-azure-classic-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/discovery-azure-classic-7.0.0-alpha1-SNAPSHOT.zip [ ] 1%?? [> ] 2%?? [> ] 3%?? [=> ] 4%?? [=> ] 5%?? [==> ] 6%?? [==> ] 7%?? [===> ] 8%?? [===> ] 9%?? [====> ] 10%?? [====> ] 11%?? [=====> ] 12%?? [=====> ] 13%?? [======> ] 14%?? [======> ] 15%?? [=======> ] 16%?? [=======> ] 17%?? [========> ] 18%?? [========> ] 19%?? [=========> ] 20%?? [=========> ] 21%?? [==========> ] 22%?? [==========> ] 23%?? [===========> ] 24%?? [===========> ] 25%?? [============> ] 26%?? [============> ] 27%?? [=============> ] 28%?? [=============> ] 29%?? [==============> ] 30%?? [==============> ] 31%?? [===============> ] 32%?? [===============> ] 33%?? [================> ] 34%?? [================> ] 35%?? [=================> ] 36%?? [=================> ] 37%?? [==================> ] 38%?? [==================> ] 39%?? [===================> ] 40%?? [===================> ] 41%?? [====================> ] 42%?? [====================> ] 43%?? [=====================> ] 44%?? [=====================> ] 45%?? [======================> ] 46%?? [======================> ] 47%?? [=======================> ] 48%?? [=======================> ] 49%?? [========================> ] 50%?? [========================> ] 51%?? [=========================> ] 52%?? [=========================> ] 53%?? [==========================> ] 54%?? [==========================> ] 55%?? [===========================> ] 56%?? [===========================> ] 57%?? [============================> ] 58%?? [============================> ] 59%?? [=============================> ] 60%?? [=============================> ] 61%?? [==============================> ] 62%?? [==============================> ] 63%?? [===============================> ] 64%?? [===============================> ] 65%?? [================================> ] 66%?? [================================> ] 67%?? [=================================> ] 68%?? [=================================> ] 69%?? [==================================> ] 70%?? [==================================> ] 71%?? [===================================> ] 72%?? [===================================> ] 73%?? [====================================> ] 74%?? [====================================> ] 75%?? [=====================================> ] 76%?? [=====================================> ] 77%?? [======================================> ] 78%?? [======================================> ] 79%?? [=======================================> ] 80%?? [=======================================> ] 81%?? [========================================> ] 82%?? [========================================> ] 83%?? [=========================================> ] 84%?? [=========================================> ] 85%?? [==========================================> ] 86%?? [==========================================> ] 87%?? [===========================================> ] 88%?? [===========================================> ] 89%?? [============================================> ] 90%?? [============================================> ] 91%?? [=============================================> ] 92%?? [=============================================> ] 93%?? [==============================================> ] 94%?? [==============================================> ] 95%?? [===============================================> ] 96%?? [===============================================> ] 97%?? [================================================>] 98%?? [================================================>] 99%?? [=================================================] 100%?? @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.net.SocketPermission * connect,resolve See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed discovery-azure-classic :docs:integTestCluster#installDiscoveryAzureClassicPlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.312 secs. :docs:integTestCluster#installDiscoveryEc2Plugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installDiscoveryEc2Plugin Task ':docs:integTestCluster#installDiscoveryEc2Plugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/discovery-ec2-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/discovery-ec2-7.0.0-alpha1-SNAPSHOT.zip [ ] 1%?? [> ] 2%?? [> ] 3%?? [=> ] 4%?? [=> ] 5%?? [==> ] 6%?? [==> ] 7%?? [===> ] 8%?? [===> ] 9%?? [====> ] 10%?? [====> ] 11%?? [=====> ] 12%?? [=====> ] 13%?? [======> ] 14%?? [======> ] 15%?? [=======> ] 16%?? [=======> ] 17%?? [========> ] 18%?? [========> ] 19%?? [=========> ] 20%?? [=========> ] 21%?? [==========> ] 22%?? [==========> ] 23%?? [===========> ] 24%?? [===========> ] 25%?? [============> ] 26%?? [============> ] 27%?? [=============> ] 28%?? [=============> ] 29%?? [==============> ] 30%?? [==============> ] 31%?? [===============> ] 32%?? [===============> ] 33%?? [================> ] 34%?? [================> ] 35%?? [=================> ] 36%?? [=================> ] 37%?? [==================> ] 38%?? [==================> ] 39%?? [===================> ] 40%?? [===================> ] 41%?? [====================> ] 42%?? [====================> ] 43%?? [=====================> ] 44%?? [=====================> ] 45%?? [======================> ] 46%?? [======================> ] 47%?? [=======================> ] 48%?? [=======================> ] 49%?? [========================> ] 50%?? [========================> ] 51%?? [=========================> ] 52%?? [=========================> ] 53%?? [==========================> ] 54%?? [==========================> ] 55%?? [===========================> ] 56%?? [===========================> ] 57%?? [============================> ] 58%?? [============================> ] 59%?? [=============================> ] 60%?? [=============================> ] 61%?? [==============================> ] 62%?? [==============================> ] 63%?? [===============================> ] 64%?? [===============================> ] 65%?? [================================> ] 66%?? [================================> ] 67%?? [=================================> ] 68%?? [=================================> ] 69%?? [==================================> ] 70%?? [==================================> ] 71%?? [===================================> ] 72%?? [===================================> ] 73%?? [====================================> ] 74%?? [====================================> ] 75%?? [=====================================> ] 76%?? [=====================================> ] 77%?? [======================================> ] 78%?? [======================================> ] 79%?? [=======================================> ] 80%?? [=======================================> ] 81%?? [========================================> ] 82%?? [========================================> ] 83%?? [=========================================> ] 84%?? [=========================================> ] 85%?? [==========================================> ] 86%?? [==========================================> ] 87%?? [===========================================> ] 88%?? [===========================================> ] 89%?? [============================================> ] 90%?? [============================================> ] 91%?? [=============================================> ] 92%?? [=============================================> ] 93%?? [==============================================> ] 94%?? [==============================================> ] 95%?? [===============================================> ] 96%?? [===============================================> ] 97%?? [================================================>] 98%?? [================================================>] 99%?? [=================================================] 100%?? @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission accessDeclaredMembers * java.lang.RuntimePermission getClassLoader * java.net.SocketPermission * connect,resolve See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed discovery-ec2 :docs:integTestCluster#installDiscoveryEc2Plugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.266 secs. :docs:integTestCluster#installDiscoveryFilePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installDiscoveryFilePlugin Task ':docs:integTestCluster#installDiscoveryFilePlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/discovery-file-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/discovery-file-7.0.0-alpha1-SNAPSHOT.zip [=========================> ] 52%?? [================================================>] 99%?? [=================================================] 100%?? -> Installed discovery-file :docs:integTestCluster#installDiscoveryFilePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.133 secs. :docs:integTestCluster#installDiscoveryGcePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installDiscoveryGcePlugin Task ':docs:integTestCluster#installDiscoveryGcePlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/discovery-gce-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/discovery-gce-7.0.0-alpha1-SNAPSHOT.zip [ ] 1%?? [> ] 2%?? [> ] 3%?? [=> ] 4%?? [=> ] 5%?? [==> ] 6%?? [==> ] 7%?? [===> ] 8%?? [===> ] 9%?? [====> ] 10%?? [====> ] 11%?? [=====> ] 12%?? [=====> ] 13%?? [======> ] 14%?? [======> ] 15%?? [=======> ] 16%?? [=======> ] 17%?? [========> ] 18%?? [========> ] 19%?? [=========> ] 20%?? [=========> ] 21%?? [==========> ] 22%?? [==========> ] 23%?? [===========> ] 24%?? [===========> ] 25%?? [============> ] 26%?? [============> ] 27%?? [=============> ] 28%?? [=============> ] 29%?? [==============> ] 30%?? [==============> ] 31%?? [===============> ] 32%?? [===============> ] 33%?? [================> ] 34%?? [================> ] 35%?? [=================> ] 36%?? [=================> ] 37%?? [==================> ] 38%?? [==================> ] 39%?? [===================> ] 40%?? [===================> ] 41%?? [====================> ] 42%?? [====================> ] 43%?? [=====================> ] 44%?? [=====================> ] 45%?? [======================> ] 46%?? [======================> ] 47%?? [=======================> ] 48%?? [=======================> ] 49%?? [========================> ] 50%?? [========================> ] 51%?? [=========================> ] 52%?? [=========================> ] 53%?? [==========================> ] 54%?? [==========================> ] 55%?? [===========================> ] 56%?? [===========================> ] 57%?? [============================> ] 58%?? [============================> ] 59%?? [=============================> ] 60%?? [=============================> ] 61%?? [==============================> ] 62%?? [==============================> ] 63%?? [===============================> ] 64%?? [===============================> ] 65%?? [================================> ] 66%?? [================================> ] 67%?? [=================================> ] 68%?? [=================================> ] 69%?? [==================================> ] 70%?? [==================================> ] 71%?? [===================================> ] 72%?? [===================================> ] 73%?? [====================================> ] 74%?? [====================================> ] 75%?? [=====================================> ] 76%?? [=====================================> ] 77%?? [======================================> ] 78%?? [======================================> ] 79%?? [=======================================> ] 80%?? [=======================================> ] 81%?? [========================================> ] 82%?? [========================================> ] 83%?? [=========================================> ] 84%?? [=========================================> ] 85%?? [==========================================> ] 86%?? [==========================================> ] 87%?? [===========================================> ] 88%?? [===========================================> ] 89%?? [============================================> ] 90%?? [============================================> ] 91%?? [=============================================> ] 92%?? [=============================================> ] 93%?? [==============================================> ] 94%?? [==============================================> ] 95%?? [===============================================> ] 96%?? [===============================================> ] 97%?? [================================================>] 98%?? [================================================>] 99%?? [=================================================] 100%?? @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission accessDeclaredMembers * java.lang.RuntimePermission setFactory * java.lang.reflect.ReflectPermission suppressAccessChecks * java.net.SocketPermission * connect,resolve See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed discovery-gce :docs:integTestCluster#installDiscoveryGcePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.405 secs. :docs:integTestCluster#installIngestAttachmentPlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installIngestAttachmentPlugin Task ':docs:integTestCluster#installIngestAttachmentPlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/ingest-attachment-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/ingest-attachment-7.0.0-alpha1-SNAPSHOT.zip [ ] 1%?? [> ] 2%?? [> ] 3%?? [=> ] 4%?? [=> ] 5%?? [==> ] 6%?? [==> ] 7%?? [===> ] 8%?? [===> ] 9%?? [====> ] 10%?? [====> ] 11%?? [=====> ] 12%?? [=====> ] 13%?? [======> ] 14%?? [======> ] 15%?? [=======> ] 16%?? [=======> ] 17%?? [========> ] 18%?? [========> ] 19%?? [=========> ] 20%?? [=========> ] 21%?? [==========> ] 22%?? [==========> ] 23%?? [===========> ] 24%?? [===========> ] 25%?? [============> ] 26%?? [============> ] 27%?? [=============> ] 28%?? [=============> ] 29%?? [==============> ] 30%?? [==============> ] 31%?? [===============> ] 32%?? [===============> ] 33%?? [================> ] 34%?? [================> ] 35%?? [=================> ] 36%?? [=================> ] 37%?? [==================> ] 38%?? [==================> ] 39%?? [===================> ] 40%?? [===================> ] 41%?? [====================> ] 42%?? [====================> ] 43%?? [=====================> ] 44%?? [=====================> ] 45%?? [======================> ] 46%?? [======================> ] 47%?? [=======================> ] 48%?? [=======================> ] 49%?? [========================> ] 50%?? [========================> ] 51%?? [=========================> ] 52%?? [=========================> ] 53%?? [==========================> ] 54%?? [==========================> ] 55%?? [===========================> ] 56%?? [===========================> ] 57%?? [============================> ] 58%?? [============================> ] 59%?? [=============================> ] 60%?? [=============================> ] 61%?? [==============================> ] 62%?? [==============================> ] 63%?? [===============================> ] 64%?? [===============================> ] 65%?? [================================> ] 66%?? [================================> ] 67%?? [=================================> ] 68%?? [=================================> ] 69%?? [==================================> ] 70%?? [==================================> ] 71%?? [===================================> ] 72%?? [===================================> ] 73%?? [====================================> ] 74%?? [====================================> ] 75%?? [=====================================> ] 76%?? [=====================================> ] 77%?? [======================================> ] 78%?? [======================================> ] 79%?? [=======================================> ] 80%?? [=======================================> ] 81%?? [========================================> ] 82%?? [========================================> ] 83%?? [=========================================> ] 84%?? [=========================================> ] 85%?? [==========================================> ] 86%?? [==========================================> ] 87%?? [===========================================> ] 88%?? [===========================================> ] 89%?? [============================================> ] 90%?? [============================================> ] 91%?? [=============================================> ] 92%?? [=============================================> ] 93%?? [==============================================> ] 94%?? [==============================================> ] 95%?? [===============================================> ] 96%?? [===============================================> ] 97%?? [================================================>] 98%?? [================================================>] 99%?? [=================================================] 100%?? @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission accessDeclaredMembers * java.lang.RuntimePermission getClassLoader * java.lang.reflect.ReflectPermission suppressAccessChecks * java.security.SecurityPermission createAccessControlContext * java.security.SecurityPermission insertProvider * java.security.SecurityPermission putProviderProperty.BC See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed ingest-attachment :docs:integTestCluster#installIngestAttachmentPlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.808 secs. :docs:integTestCluster#installIngestGeoipPlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installIngestGeoipPlugin Task ':docs:integTestCluster#installIngestGeoipPlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/ingest-geoip-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/ingest-geoip-7.0.0-alpha1-SNAPSHOT.zip [ ] 1%?? [> ] 2%?? [> ] 3%?? [=> ] 4%?? [=> ] 5%?? [==> ] 6%?? [==> ] 7%?? [===> ] 8%?? [===> ] 9%?? [====> ] 10%?? [====> ] 11%?? [=====> ] 12%?? [=====> ] 13%?? [======> ] 14%?? [======> ] 15%?? [=======> ] 16%?? [=======> ] 17%?? [========> ] 18%?? [========> ] 19%?? [=========> ] 20%?? [=========> ] 21%?? [==========> ] 22%?? [==========> ] 23%?? [===========> ] 24%?? [===========> ] 25%?? [============> ] 26%?? [============> ] 27%?? [=============> ] 28%?? [=============> ] 29%?? [==============> ] 30%?? [==============> ] 31%?? [===============> ] 32%?? [===============> ] 33%?? [================> ] 34%?? [================> ] 35%?? [=================> ] 36%?? [=================> ] 37%?? [==================> ] 38%?? [==================> ] 39%?? [===================> ] 40%?? [===================> ] 41%?? [====================> ] 42%?? [====================> ] 43%?? [=====================> ] 44%?? [=====================> ] 45%?? [======================> ] 46%?? [======================> ] 47%?? [=======================> ] 48%?? [=======================> ] 49%?? [========================> ] 50%?? [========================> ] 51%?? [=========================> ] 52%?? [=========================> ] 53%?? [==========================> ] 54%?? [==========================> ] 55%?? [===========================> ] 56%?? [===========================> ] 57%?? [============================> ] 58%?? [============================> ] 59%?? [=============================> ] 60%?? [=============================> ] 61%?? [==============================> ] 62%?? [==============================> ] 63%?? [===============================> ] 64%?? [===============================> ] 65%?? [================================> ] 66%?? [================================> ] 67%?? [=================================> ] 68%?? [=================================> ] 69%?? [==================================> ] 70%?? [==================================> ] 71%?? [===================================> ] 72%?? [===================================> ] 73%?? [====================================> ] 74%?? [====================================> ] 75%?? [=====================================> ] 76%?? [=====================================> ] 77%?? [======================================> ] 78%?? [======================================> ] 79%?? [=======================================> ] 80%?? [=======================================> ] 81%?? [========================================> ] 82%?? [========================================> ] 83%?? [=========================================> ] 84%?? [=========================================> ] 85%?? [==========================================> ] 86%?? [==========================================> ] 87%?? [===========================================> ] 88%?? [===========================================> ] 89%?? [============================================> ] 90%?? [============================================> ] 91%?? [=============================================> ] 92%?? [=============================================> ] 93%?? [==============================================> ] 94%?? [==============================================> ] 95%?? [===============================================> ] 96%?? [===============================================> ] 97%?? [================================================>] 98%?? [================================================>] 99%?? [=================================================] 100%?? @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission accessDeclaredMembers * java.lang.reflect.ReflectPermission suppressAccessChecks See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. -> Installed ingest-geoip :docs:integTestCluster#installIngestGeoipPlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 3.4 secs. :docs:integTestCluster#installIngestUserAgentPlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installIngestUserAgentPlugin Task ':docs:integTestCluster#installIngestUserAgentPlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/ingest-user-agent-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/ingest-user-agent-7.0.0-alpha1-SNAPSHOT.zip [=====> ] 13%?? [============> ] 26%?? [==================> ] 39%?? [=========================> ] 52%?? [===============================> ] 65%?? [======================================> ] 78%?? [============================================> ] 91%?? [================================================>] 99%?? [=================================================] 100%?? -> Installed ingest-user-agent :docs:integTestCluster#installIngestUserAgentPlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.314 secs. :docs:integTestCluster#installMapperMurmur3Plugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installMapperMurmur3Plugin Task ':docs:integTestCluster#installMapperMurmur3Plugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/mapper-murmur3-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/mapper-murmur3-7.0.0-alpha1-SNAPSHOT.zip [========================> ] 51%?? [================================================>] 99%?? [=================================================] 100%?? -> Installed mapper-murmur3 :docs:integTestCluster#installMapperMurmur3Plugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.285 secs. :docs:integTestCluster#installMapperSizePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installMapperSizePlugin Task ':docs:integTestCluster#installMapperSizePlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/mapper-size-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/mapper-size-7.0.0-alpha1-SNAPSHOT.zip [======================> ] 47%?? [==============================================> ] 95%?? [================================================>] 99%?? [=================================================] 100%?? -> Installed mapper-size :docs:integTestCluster#installMapperSizePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.406 secs. :docs:integTestCluster#installStoreSmbPlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installStoreSmbPlugin Task ':docs:integTestCluster#installStoreSmbPlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/store-smb-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/store-smb-7.0.0-alpha1-SNAPSHOT.zip [=========================> ] 52%?? [================================================>] 99%?? [=================================================] 100%?? -> Installed store-smb :docs:integTestCluster#installStoreSmbPlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.414 secs. :docs:integTestCluster#installTransportNioPlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#installTransportNioPlugin Task ':docs:integTestCluster#installTransportNioPlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/cluster/integTestCluster%20node0/plugins%20tmp/transport-nio-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/docs/build/cluster/integTestCluster node0/plugins tmp/transport-nio-7.0.0-alpha1-SNAPSHOT.zip [=====> ] 12%?? [===========> ] 24%?? [=================> ] 36%?? [=======================> ] 48%?? [=============================> ] 60%?? [===================================> ] 72%?? [=========================================> ] 84%?? [===============================================> ] 96%?? [================================================>] 99%?? [=================================================] 100%?? -> Installed transport-nio :docs:integTestCluster#installTransportNioPlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.442 secs. :docs:integTestCluster#extraConfig (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#extraConfig Task ':docs:integTestCluster#extraConfig' is not up-to-date because: No history is available. :docs:integTestCluster#extraConfig (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.294 secs. :docs:integTestCluster#start (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#start Task ':docs:integTestCluster#start' is not up-to-date because: Task has not declared any outputs despite executing actions. Node 0 configuration: |----------------------------------------- | cwd: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\cwd | command: cmd /C " C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\cwd\run.bat -E node.portsfile=true -E path.data=C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\data " | environment: | JAVA_HOME: C:\Users\jenkins\.java\java8 | ES_JAVA_OPTS: -ea -esa -Xms512m -Xmx512m | ES_PATH_CONF: C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\config | | [run.bat] "C:\Users\jenkins\WORKSP~1\ELASTI~1\docs\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch.bat" %* > run.log 2>&1 if "%errorlevel%" neq "0" ( type nul >> run.failed ) | | [elasticsearch.yml] | cluster.name: docs_integTestCluster | node.name: node-0 | pidfile: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\integTestCluster node0\es.pid | path.repo: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\shared/repo | path.shared_data: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\cluster\shared/ | node.attr.testattr: test | node.max_local_storage_nodes: 1 | http.port: 0 | transport.tcp.port: 0 | cluster.routing.allocation.disk.watermark.low: 1b | cluster.routing.allocation.disk.watermark.high: 1b | cluster.routing.allocation.disk.watermark.flood_stage: 1b | script.max_compilations_rate: 2048/1m | reindex.remote.whitelist: 127.0.0.1:* | script.painless.regex.enabled: true |----------------------------------------- :docs:integTestCluster#start (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.042 secs. :rest-api-spec:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :rest-api-spec:compileJava NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\rest-api-spec\src\main\java', not found Skipping task ':rest-api-spec:compileJava' as it has no source files and no previous output files. :rest-api-spec:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :rest-api-spec:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :rest-api-spec:processResources Task ':rest-api-spec:processResources' is not up-to-date because: No history is available. :rest-api-spec:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.763 secs. :rest-api-spec:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :rest-api-spec:classes Skipping task ':rest-api-spec:classes' as it has no actions. :rest-api-spec:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :rest-api-spec:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :rest-api-spec:jar Task ':rest-api-spec:jar' is not up-to-date because: No history is available. :rest-api-spec:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.168 secs. :docs:copyRestSpec (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:copyRestSpec Task ':docs:copyRestSpec' is not up-to-date because: No history is available. :docs:copyRestSpec (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.164 secs. :docs:checkstyle (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:checkstyle Skipping task ':docs:checkstyle' as it has no actions. :docs:checkstyle (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :docs:filepermissions (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:filepermissions SKIPPED Skipping task ':docs:filepermissions' as task onlyIf is false. :docs:filepermissions (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :docs:forbiddenPatterns (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:forbiddenPatterns Task ':docs:forbiddenPatterns' is not up-to-date because: No history is available. :docs:forbiddenPatterns (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.021 secs. :docs:jarHell (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:jarHell Task ':docs:jarHell' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs Command: C:\Users\jenkins\.java\java8\bin\java -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\resources\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\rest;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar org.elasticsearch.bootstrap.JarHell Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' checking for jar hell... java.class.path: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\resources\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\rest;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar sun.boot.class.path: C:\Users\jenkins\.java\java8\jre\lib\resources.jar;C:\Users\jenkins\.java\java8\jre\lib\rt.jar;C:\Users\jenkins\.java\java8\jre\lib\sunrsasign.jar;C:\Users\jenkins\.java\java8\jre\lib\jsse.jar;C:\Users\jenkins\.java\java8\jre\lib\jce.jar;C:\Users\jenkins\.java\java8\jre\lib\charsets.jar;C:\Users\jenkins\.java\java8\jre\lib\jfr.jar;C:\Users\jenkins\.java\java8\jre\classes classloader urls: [file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/classes/java/test/, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/resources/test/, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/docs/build/rest/, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/test/framework/build/distributions/framework-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/client/rest/build/distributions/elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/elasticsearch-nio/build/distributions/elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/server/build/distributions/elasticsearch-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/server/cli/build/distributions/elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.carrotsearch.randomizedtesting/randomizedtesting-runner/2.5.2/91f3284993b44dcb2f003b5f28617abba13971d2/randomizedtesting-runner-2.5.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/junit/junit/4.12/2973d150c0dc1fefe998f834810d68f278ea58ec/junit-4.12.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-all/1.3/63a21ebc981131004ad02e0434e799fd7f3a8d5a/hamcrest-all-1.3.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-test-framework/7.4.0-snapshot-1ed95c097b/1afd168832ee0cf36d1adcc960c3e127d8848bd0/lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-codecs/7.4.0-snapshot-1ed95c097b/4ad08cbda4541cbf5899860aa99c08e557585f74/lucene-codecs-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/commons-logging/commons-logging/1.1.3/f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f/commons-logging-1.1.3.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/commons-codec/commons-codec/1.10/4b95f4897fa13f2cd904aee711aeafc0c5295cd8/commons-codec-1.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.elasticsearch/securemock/1.2/98201d4ad5ac93f6b415ae9172d52b5e7cda490e/securemock-1.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.elasticsearch/mocksocket/1.2/190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361/mocksocket-1.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpclient/4.5.2/733db77aa8d9b2d68015189df76ab06304406e50/httpclient-4.5.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpcore/4.4.5/e7501a1b34325abb00d17dde96150604a0658b54/httpcore-4.4.5.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpasyncclient/4.1.2/95aa3e6fb520191a0970a73cf09f62948ee614be/httpasyncclient-4.1.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpcore-nio/4.4.5/f4be009e7505f6ceddf21e7960c759f413f15056/httpcore-nio-4.4.5.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.9.1/7a2999229464e7a324aa503c0a52ec0f05efe7bd/log4j-api-2.9.1.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/x-content/build/distributions/elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/elasticsearch-core/build/distributions/elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/libs/secure-sm/build/distributions/elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/7.4.0-snapshot-1ed95c097b/b06e4f6514256a3f187a9892e520638b9c59e63/lucene-core-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/7.4.0-snapshot-1ed95c097b/2b2be48f6622c150496e755497e7bdb8daa46030/lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-backward-codecs/7.4.0-snapshot-1ed95c097b/6cbafc48e8ac4966377665eb3bbe93f9addf04a5/lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-grouping/7.4.0-snapshot-1ed95c097b/4c71cef87fe513a7a96c2a7980ed6f7c2b015763/lucene-grouping-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-highlighter/7.4.0-snapshot-1ed95c097b/665e044d1180100940bccd7e8e41dde48e342da3/lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-join/7.4.0-snapshot-1ed95c097b/d343bbf5792f5969288b59b51179acd29d04f4ee/lucene-join-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-memory/7.4.0-snapshot-1ed95c097b/8915f3c93af3348655bcc204289f9011835738a2/lucene-memory-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-misc/7.4.0-snapshot-1ed95c097b/e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d/lucene-misc-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queries/7.4.0-snapshot-1ed95c097b/5946d5e2be276f66e9ff6d6111acabb03a9330d9/lucene-queries-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/7.4.0-snapshot-1ed95c097b/d9fc5fc63f3d861e5af72e11373368e8a4c6bba6/lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/7.4.0-snapshot-1ed95c097b/ee283c0a1a717f3e0915de75864a93d043efaee3/lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial/7.4.0-snapshot-1ed95c097b/e1adf0220a7c052ac81e2919ffac24ac0e5b007c/lucene-spatial-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial-extras/7.4.0-snapshot-1ed95c097b/6d9306053942c48f43392a634f11a95462b5996e/lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial3d/7.4.0-snapshot-1ed95c097b/2334e8c5f4d0f98659b30e0c2035296e4aae8ff5/lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-suggest/7.4.0-snapshot-1ed95c097b/f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463/lucene-suggest-7.4.0-snapshot-1ed95c097b.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.carrotsearch/hppc/0.7.1/8b5057f74ea378c0150a1860874a3ebdcb713767/hppc-0.7.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/joda-time/joda-time/2.9.9/f7b520c458572890807d143670c9b24f4de90897/joda-time-2.9.9.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.tdunning/t-digest/3.2/2ab94758b0276a8a26102adf8d528cf6d0567b9a/t-digest-3.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.hdrhistogram/HdrHistogram/2.1.9/e4631ce165eb400edecfa32e03d3f1be53dee754/HdrHistogram-2.1.9.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.locationtech.spatial4j/spatial4j/0.7/faa8ba85d503da4ab872d17ba8c00da0098ab2f2/spatial4j-0.7.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.locationtech.jts/jts-core/1.15.0/705981b7e25d05a76a3654e597dab6ba423eb79e/jts-core-1.15.0.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.9.1/c041978c686866ee8534f538c6220238db3bb6be/log4j-core-2.9.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-1.2-api/2.9.1/894f96d677880d4ab834a1356f62b875e579caaa/log4j-1.2-api-2.9.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.elasticsearch/jna/4.5.1/da10908ae23dc59b19dc258e63aea1c44621dc3a/jna-4.5.1.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/5.0.2/98cafc6081d5632b61be2c9e60650b64ddbc637c/jopt-simple-5.0.2.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/org.yaml/snakeyaml/1.17/7a27ea250c5130b2922b86dea63cbb1cc10a660c/snakeyaml-1.17.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.8.10/eb21a035c66ad307e66ec8fce37f5d50fd62d039/jackson-core-2.8.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-smile/2.8.10/e853081fadaad3e98ed801937acc3d8f77580686/jackson-dataformat-smile-2.8.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-yaml/2.8.10/1e08caf1d787c825307d8cc6362452086020d853/jackson-dataformat-yaml-2.8.10.jar, file:/C:/Users/jenkins.SLAVE-0633DD87F/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-cbor/2.8.10/1c58cc9313ddf19f0900cd61ed044874278ce320/jackson-dataformat-cbor-2.8.10.jar] java.home: C:\Users\jenkins\.java\java8\jre examining directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\classes\java\test examining directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\resources\test examining directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\rest examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar examining jar: C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar no jar hell found :docs:jarHell (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.684 secs. :docs:licenseHeaders (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:licenseHeaders Task ':docs:licenseHeaders' is not up-to-date because: No history is available. :docs:licenseHeaders (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.339 secs. :docs:loggerUsageCheck (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:loggerUsageCheck Task ':docs:loggerUsageCheck' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs Command: C:\Users\jenkins\.java\java8\bin\java -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\logger-usage\build\distributions\logger-usage-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.ow2.asm\asm-debug-all\5.0.4\702b8525fcf81454235e5e2fa2a35f15ffc0ec7e\asm-debug-all-5.0.4.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar org.elasticsearch.test.loggerusage.ESLoggerUsageChecker C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\classes\java\test Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' checking for wrong usages of ESLogger... No wrong usages found :docs:loggerUsageCheck (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.49 secs. :docs:namingConventions (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:namingConventions Task ':docs:namingConventions' is not up-to-date because: No history is available. Starting process 'command 'C:\Users\jenkins\.java\java8\bin\java''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs Command: C:\Users\jenkins\.java\java8\bin\java -Djna.nosys=true -cp C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\buildSrc\build\distributions\build-tools-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\test\framework\build\distributions\framework-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\rest\build\distributions\elasticsearch-rest-client-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-nio\build\distributions\elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\distributions\elasticsearch-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\cli\build\distributions\elasticsearch-cli-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch.randomizedtesting\randomizedtesting-runner\2.5.2\91f3284993b44dcb2f003b5f28617abba13971d2\randomizedtesting-runner-2.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-all\1.3\63a21ebc981131004ad02e0434e799fd7f3a8d5a\hamcrest-all-1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-test-framework\7.4.0-snapshot-1ed95c097b\1afd168832ee0cf36d1adcc960c3e127d8848bd0\lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-codecs\7.4.0-snapshot-1ed95c097b\4ad08cbda4541cbf5899860aa99c08e557585f74\lucene-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-logging\commons-logging\1.1.3\f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f\commons-logging-1.1.3.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\commons-codec\commons-codec\1.10\4b95f4897fa13f2cd904aee711aeafc0c5295cd8\commons-codec-1.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\securemock\1.2\98201d4ad5ac93f6b415ae9172d52b5e7cda490e\securemock-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\mocksocket\1.2\190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361\mocksocket-1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient\4.5.2\733db77aa8d9b2d68015189df76ab06304406e50\httpclient-4.5.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore\4.4.5\e7501a1b34325abb00d17dde96150604a0658b54\httpcore-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpasyncclient\4.1.2\95aa3e6fb520191a0970a73cf09f62948ee614be\httpasyncclient-4.1.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpcore-nio\4.4.5\f4be009e7505f6ceddf21e7960c759f413f15056\httpcore-nio-4.4.5.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.9.1\7a2999229464e7a324aa503c0a52ec0f05efe7bd\log4j-api-2.9.1.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\x-content\build\distributions\elasticsearch-x-content-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\elasticsearch-core\build\distributions\elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\libs\secure-sm\build\distributions\elasticsearch-secure-sm-7.0.0-alpha1-SNAPSHOT.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-core\7.4.0-snapshot-1ed95c097b\b06e4f6514256a3f187a9892e520638b9c59e63\lucene-core-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-analyzers-common\7.4.0-snapshot-1ed95c097b\2b2be48f6622c150496e755497e7bdb8daa46030\lucene-analyzers-common-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-backward-codecs\7.4.0-snapshot-1ed95c097b\6cbafc48e8ac4966377665eb3bbe93f9addf04a5\lucene-backward-codecs-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-grouping\7.4.0-snapshot-1ed95c097b\4c71cef87fe513a7a96c2a7980ed6f7c2b015763\lucene-grouping-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-highlighter\7.4.0-snapshot-1ed95c097b\665e044d1180100940bccd7e8e41dde48e342da3\lucene-highlighter-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-join\7.4.0-snapshot-1ed95c097b\d343bbf5792f5969288b59b51179acd29d04f4ee\lucene-join-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-memory\7.4.0-snapshot-1ed95c097b\8915f3c93af3348655bcc204289f9011835738a2\lucene-memory-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-misc\7.4.0-snapshot-1ed95c097b\e7dc67b42eca3b1546a36370b6dcda0f83b2eb7d\lucene-misc-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queries\7.4.0-snapshot-1ed95c097b\5946d5e2be276f66e9ff6d6111acabb03a9330d9\lucene-queries-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-queryparser\7.4.0-snapshot-1ed95c097b\d9fc5fc63f3d861e5af72e11373368e8a4c6bba6\lucene-queryparser-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-sandbox\7.4.0-snapshot-1ed95c097b\ee283c0a1a717f3e0915de75864a93d043efaee3\lucene-sandbox-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial\7.4.0-snapshot-1ed95c097b\e1adf0220a7c052ac81e2919ffac24ac0e5b007c\lucene-spatial-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial-extras\7.4.0-snapshot-1ed95c097b\6d9306053942c48f43392a634f11a95462b5996e\lucene-spatial-extras-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-spatial3d\7.4.0-snapshot-1ed95c097b\2334e8c5f4d0f98659b30e0c2035296e4aae8ff5\lucene-spatial3d-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.lucene\lucene-suggest\7.4.0-snapshot-1ed95c097b\f2b2c454eb7b5d73b9df1390ea4730ce3dd4e463\lucene-suggest-7.4.0-snapshot-1ed95c097b.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.carrotsearch\hppc\0.7.1\8b5057f74ea378c0150a1860874a3ebdcb713767\hppc-0.7.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\joda-time\joda-time\2.9.9\f7b520c458572890807d143670c9b24f4de90897\joda-time-2.9.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.tdunning\t-digest\3.2\2ab94758b0276a8a26102adf8d528cf6d0567b9a\t-digest-3.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.hdrhistogram\HdrHistogram\2.1.9\e4631ce165eb400edecfa32e03d3f1be53dee754\HdrHistogram-2.1.9.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.spatial4j\spatial4j\0.7\faa8ba85d503da4ab872d17ba8c00da0098ab2f2\spatial4j-0.7.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.locationtech.jts\jts-core\1.15.0\705981b7e25d05a76a3654e597dab6ba423eb79e\jts-core-1.15.0.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-core\2.9.1\c041978c686866ee8534f538c6220238db3bb6be\log4j-core-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-1.2-api\2.9.1\894f96d677880d4ab834a1356f62b875e579caaa\log4j-1.2-api-2.9.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.elasticsearch\jna\4.5.1\da10908ae23dc59b19dc258e63aea1c44621dc3a\jna-4.5.1.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\net.sf.jopt-simple\jopt-simple\5.0.2\98cafc6081d5632b61be2c9e60650b64ddbc637c\jopt-simple-5.0.2.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.17\7a27ea250c5130b2922b86dea63cbb1cc10a660c\snakeyaml-1.17.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.8.10\eb21a035c66ad307e66ec8fce37f5d50fd62d039\jackson-core-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-smile\2.8.10\e853081fadaad3e98ed801937acc3d8f77580686\jackson-dataformat-smile-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-yaml\2.8.10\1e08caf1d787c825307d8cc6362452086020d853\jackson-dataformat-yaml-2.8.10.jar;C:\Users\jenkins.SLAVE-0633DD87F\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.dataformat\jackson-dataformat-cbor\2.8.10\1c58cc9313ddf19f0900cd61ed044874278ce320\jackson-dataformat-cbor-2.8.10.jar;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\classes\java\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\resources\test;C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\rest org.elasticsearch.test.NamingConventionsCheck --test-class org.apache.lucene.util.LuceneTestCase --integ-test-class org.elasticsearch.test.ESIntegTestCase -- C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs\build\classes\java\test Successfully started process 'command 'C:\Users\jenkins\.java\java8\bin\java'' :docs:namingConventions (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.415 secs. :docs:thirdPartyAudit (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:thirdPartyAudit SKIPPED Skipping task ':docs:thirdPartyAudit' as task onlyIf is false. :docs:thirdPartyAudit (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :docs:precommit (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:precommit Skipping task ':docs:precommit' as it has no actions. :docs:precommit (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:copyPluginPropertiesTemplate Task ':x-pack:plugin:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :x-pack:plugin:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :x-pack:plugin:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:pluginProperties Task ':x-pack:plugin:pluginProperties' is not up-to-date because: No history is available. :x-pack:plugin:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.047 secs. :x-pack:plugin:core:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:core:generateNotice Task ':x-pack:plugin:core:generateNotice' is not up-to-date because: No history is available. :x-pack:plugin:core:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.018 secs. :x-pack:plugin:core:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:core:compileJava Task ':x-pack:plugin:core:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':x-pack:plugin:core:compileJava'. :x-pack:plugin:core:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\core Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\core\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. Class dependency analysis for incremental compilation took 0.483 secs. Created jar classpath snapshot for incremental compilation in 0.761 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :x-pack:plugin:core:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 20.145 secs. :x-pack:plugin:core:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:core:processResources Task ':x-pack:plugin:core:processResources' is not up-to-date because: No history is available. :x-pack:plugin:core:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.032 secs. :x-pack:plugin:core:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:core:classes Skipping task ':x-pack:plugin:core:classes' as it has no actions. :x-pack:plugin:core:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:core:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:core:jar Task ':x-pack:plugin:core:jar' is not up-to-date because: No history is available. :x-pack:plugin:core:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.4 secs. :x-pack:plugin:core:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:core:copyPluginPropertiesTemplate Task ':x-pack:plugin:core:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :x-pack:plugin:core:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:core:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:core:pluginProperties Task ':x-pack:plugin:core:pluginProperties' is not up-to-date because: No history is available. :x-pack:plugin:core:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.011 secs. :x-pack:plugin:core:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:core:bundlePlugin Task ':x-pack:plugin:core:bundlePlugin' is not up-to-date because: No history is available. :x-pack:plugin:core:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.619 secs. :x-pack:plugin:deprecation:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:deprecation:generateNotice Task ':x-pack:plugin:deprecation:generateNotice' is not up-to-date because: No history is available. :x-pack:plugin:deprecation:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.002 secs. :x-pack:plugin:deprecation:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:deprecation:compileJava Task ':x-pack:plugin:deprecation:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':x-pack:plugin:deprecation:compileJava'. :x-pack:plugin:deprecation:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\deprecation Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\deprecation\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.004 secs. Created jar classpath snapshot for incremental compilation in 0.077 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :x-pack:plugin:deprecation:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.108 secs. :x-pack:plugin:deprecation:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:deprecation:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\deprecation\src\main\resources', not found Skipping task ':x-pack:plugin:deprecation:processResources' as it has no source files and no previous output files. :x-pack:plugin:deprecation:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:deprecation:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:deprecation:classes Skipping task ':x-pack:plugin:deprecation:classes' as it has no actions. :x-pack:plugin:deprecation:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:deprecation:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:deprecation:jar Task ':x-pack:plugin:deprecation:jar' is not up-to-date because: No history is available. :x-pack:plugin:deprecation:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.013 secs. :x-pack:plugin:deprecation:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:deprecation:copyPluginPropertiesTemplate Task ':x-pack:plugin:deprecation:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :x-pack:plugin:deprecation:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :x-pack:plugin:deprecation:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:deprecation:pluginProperties Task ':x-pack:plugin:deprecation:pluginProperties' is not up-to-date because: No history is available. :x-pack:plugin:deprecation:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.009 secs. :x-pack:plugin:deprecation:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:deprecation:bundlePlugin Task ':x-pack:plugin:deprecation:bundlePlugin' is not up-to-date because: No history is available. :x-pack:plugin:deprecation:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.009 secs. :x-pack:plugin:graph:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:graph:generateNotice Task ':x-pack:plugin:graph:generateNotice' is not up-to-date because: No history is available. :x-pack:plugin:graph:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :x-pack:plugin:graph:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:graph:compileJava Task ':x-pack:plugin:graph:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':x-pack:plugin:graph:compileJava'. :x-pack:plugin:graph:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\graph Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\graph\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\graph\src\main\java\org\elasticsearch\xpack\graph\action\TransportGraphExploreAction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. Class dependency analysis for incremental compilation took 0.009 secs. Created jar classpath snapshot for incremental compilation in 0.005 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :x-pack:plugin:graph:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.021 secs. :x-pack:plugin:graph:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:graph:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\graph\src\main\resources', not found Skipping task ':x-pack:plugin:graph:processResources' as it has no source files and no previous output files. :x-pack:plugin:graph:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:graph:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:graph:classes Skipping task ':x-pack:plugin:graph:classes' as it has no actions. :x-pack:plugin:graph:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:graph:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:graph:jar Task ':x-pack:plugin:graph:jar' is not up-to-date because: No history is available. :x-pack:plugin:graph:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.015 secs. :x-pack:plugin:graph:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:graph:copyPluginPropertiesTemplate Task ':x-pack:plugin:graph:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :x-pack:plugin:graph:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :x-pack:plugin:graph:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:graph:pluginProperties Task ':x-pack:plugin:graph:pluginProperties' is not up-to-date because: No history is available. :x-pack:plugin:graph:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.012 secs. :x-pack:plugin:graph:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:graph:bundlePlugin Task ':x-pack:plugin:graph:bundlePlugin' is not up-to-date because: No history is available. :x-pack:plugin:graph:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.013 secs. :x-pack:plugin:logstash:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:logstash:generateNotice Task ':x-pack:plugin:logstash:generateNotice' is not up-to-date because: No history is available. :x-pack:plugin:logstash:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.002 secs. :x-pack:plugin:logstash:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:logstash:compileJava Task ':x-pack:plugin:logstash:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':x-pack:plugin:logstash:compileJava'. :x-pack:plugin:logstash:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\logstash Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\logstash\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.002 secs. Created jar classpath snapshot for incremental compilation in 0.006 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :x-pack:plugin:logstash:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.454 secs. :x-pack:plugin:logstash:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:logstash:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\logstash\src\main\resources', not found Skipping task ':x-pack:plugin:logstash:processResources' as it has no source files and no previous output files. :x-pack:plugin:logstash:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:logstash:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:logstash:classes Skipping task ':x-pack:plugin:logstash:classes' as it has no actions. :x-pack:plugin:logstash:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:logstash:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:logstash:jar Task ':x-pack:plugin:logstash:jar' is not up-to-date because: No history is available. :x-pack:plugin:logstash:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.012 secs. :x-pack:plugin:logstash:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:logstash:copyPluginPropertiesTemplate Task ':x-pack:plugin:logstash:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :x-pack:plugin:logstash:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :x-pack:plugin:logstash:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:logstash:pluginProperties Task ':x-pack:plugin:logstash:pluginProperties' is not up-to-date because: No history is available. :x-pack:plugin:logstash:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.009 secs. :x-pack:plugin:logstash:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:logstash:bundlePlugin Task ':x-pack:plugin:logstash:bundlePlugin' is not up-to-date because: No history is available. :x-pack:plugin:logstash:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.012 secs. :x-pack:plugin:ml:cpp-snapshot:downloadMachineLearningSnapshot (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:ml:cpp-snapshot:downloadMachineLearningSnapshot Task ':x-pack:plugin:ml:cpp-snapshot:downloadMachineLearningSnapshot' is not up-to-date because: Task has not declared any outputs despite executing actions. :x-pack:plugin:ml:cpp-snapshot:downloadMachineLearningSnapshot (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 6.78 secs. :x-pack:plugin:ml:extractNativeLicenses (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:ml:extractNativeLicenses Deleting stale output file: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\ml\build Task ':x-pack:plugin:ml:extractNativeLicenses' is not up-to-date because: No history is available. :x-pack:plugin:ml:extractNativeLicenses (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.113 secs. :x-pack:plugin:ml:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:ml:generateNotice Task ':x-pack:plugin:ml:generateNotice' is not up-to-date because: Task.upToDateWhen is false. :x-pack:plugin:ml:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.008 secs. :x-pack:plugin:ml:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:ml:compileJava Task ':x-pack:plugin:ml:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':x-pack:plugin:ml:compileJava'. :x-pack:plugin:ml:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\ml Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\ml\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. Class dependency analysis for incremental compilation took 0.157 secs. Created jar classpath snapshot for incremental compilation in 0.026 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :x-pack:plugin:ml:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 7.064 secs. :x-pack:plugin:ml:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:ml:processResources Task ':x-pack:plugin:ml:processResources' is not up-to-date because: No history is available. :x-pack:plugin:ml:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.008 secs. :x-pack:plugin:ml:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:ml:classes Skipping task ':x-pack:plugin:ml:classes' as it has no actions. :x-pack:plugin:ml:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:ml:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:ml:jar Task ':x-pack:plugin:ml:jar' is not up-to-date because: No history is available. :x-pack:plugin:ml:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.177 secs. :x-pack:plugin:ml:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:ml:copyPluginPropertiesTemplate Task ':x-pack:plugin:ml:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :x-pack:plugin:ml:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :x-pack:plugin:ml:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:ml:pluginProperties Task ':x-pack:plugin:ml:pluginProperties' is not up-to-date because: No history is available. :x-pack:plugin:ml:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.013 secs. :x-pack:plugin:ml:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:ml:bundlePlugin Task ':x-pack:plugin:ml:bundlePlugin' is not up-to-date because: No history is available. :x-pack:plugin:ml:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 4.465 secs. :client:sniffer:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :client:sniffer:compileJava Task ':client:sniffer:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':client:sniffer:compileJava'. :client:sniffer:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\sniffer Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\sniffer\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.005 secs. Created jar classpath snapshot for incremental compilation in 0.001 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :client:sniffer:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.142 secs. :client:sniffer:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :client:sniffer:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\client\sniffer\src\main\resources', not found Skipping task ':client:sniffer:processResources' as it has no source files and no previous output files. :client:sniffer:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :client:sniffer:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :client:sniffer:classes Skipping task ':client:sniffer:classes' as it has no actions. :client:sniffer:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :client:sniffer:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :client:sniffer:jar Task ':client:sniffer:jar' is not up-to-date because: No history is available. :client:sniffer:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.014 secs. :x-pack:plugin:monitoring:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:monitoring:generateNotice Task ':x-pack:plugin:monitoring:generateNotice' is not up-to-date because: No history is available. :x-pack:plugin:monitoring:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.002 secs. :x-pack:plugin:monitoring:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:monitoring:compileJava Task ':x-pack:plugin:monitoring:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':x-pack:plugin:monitoring:compileJava'. :x-pack:plugin:monitoring:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\monitoring Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\monitoring\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.036 secs. Created jar classpath snapshot for incremental compilation in 0.006 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :x-pack:plugin:monitoring:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 3.427 secs. :x-pack:plugin:monitoring:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:monitoring:processResources Task ':x-pack:plugin:monitoring:processResources' is not up-to-date because: No history is available. :x-pack:plugin:monitoring:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.035 secs. :x-pack:plugin:monitoring:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:monitoring:classes Skipping task ':x-pack:plugin:monitoring:classes' as it has no actions. :x-pack:plugin:monitoring:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:monitoring:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:monitoring:jar Task ':x-pack:plugin:monitoring:jar' is not up-to-date because: No history is available. :x-pack:plugin:monitoring:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.047 secs. :x-pack:plugin:monitoring:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:monitoring:copyPluginPropertiesTemplate Task ':x-pack:plugin:monitoring:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :x-pack:plugin:monitoring:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:monitoring:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:monitoring:pluginProperties Task ':x-pack:plugin:monitoring:pluginProperties' is not up-to-date because: No history is available. :x-pack:plugin:monitoring:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.014 secs. :x-pack:plugin:monitoring:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:monitoring:bundlePlugin Task ':x-pack:plugin:monitoring:bundlePlugin' is not up-to-date because: No history is available. :x-pack:plugin:monitoring:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.029 secs. :x-pack:plugin:rollup:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:rollup:generateNotice Task ':x-pack:plugin:rollup:generateNotice' is not up-to-date because: No history is available. :x-pack:plugin:rollup:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :x-pack:plugin:rollup:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:rollup:compileJava Task ':x-pack:plugin:rollup:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':x-pack:plugin:rollup:compileJava'. :x-pack:plugin:rollup:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\rollup Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\rollup\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\rollup\src\main\java\org\elasticsearch\xpack\rollup\action\TransportRollupSearchAction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. Class dependency analysis for incremental compilation took 0.021 secs. Created jar classpath snapshot for incremental compilation in 0.005 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :x-pack:plugin:rollup:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 3.148 secs. :x-pack:plugin:rollup:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:rollup:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\rollup\src\main\resources', not found Skipping task ':x-pack:plugin:rollup:processResources' as it has no source files and no previous output files. :x-pack:plugin:rollup:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:rollup:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:rollup:classes Skipping task ':x-pack:plugin:rollup:classes' as it has no actions. :x-pack:plugin:rollup:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:rollup:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:rollup:jar Task ':x-pack:plugin:rollup:jar' is not up-to-date because: No history is available. :x-pack:plugin:rollup:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.03 secs. :x-pack:plugin:rollup:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:rollup:copyPluginPropertiesTemplate Task ':x-pack:plugin:rollup:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :x-pack:plugin:rollup:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :x-pack:plugin:rollup:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:rollup:pluginProperties Task ':x-pack:plugin:rollup:pluginProperties' is not up-to-date because: No history is available. :x-pack:plugin:rollup:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.011 secs. :x-pack:plugin:rollup:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:rollup:bundlePlugin Task ':x-pack:plugin:rollup:bundlePlugin' is not up-to-date because: No history is available. :x-pack:plugin:rollup:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.02 secs. :x-pack:plugin:security:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:security:generateNotice Task ':x-pack:plugin:security:generateNotice' is not up-to-date because: No history is available. :x-pack:plugin:security:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.006 secs. :x-pack:plugin:security:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:security:compileJava Task ':x-pack:plugin:security:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':x-pack:plugin:security:compileJava'. :x-pack:plugin:security:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\security Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\security\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. Class dependency analysis for incremental compilation took 0.221 secs. Created jar classpath snapshot for incremental compilation in 3.064 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :x-pack:plugin:security:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 11.024 secs. :x-pack:plugin:security:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:security:processResources Task ':x-pack:plugin:security:processResources' is not up-to-date because: No history is available. :x-pack:plugin:security:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.015 secs. :x-pack:plugin:security:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:security:classes Skipping task ':x-pack:plugin:security:classes' as it has no actions. :x-pack:plugin:security:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:security:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:security:jar Task ':x-pack:plugin:security:jar' is not up-to-date because: No history is available. :x-pack:plugin:security:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.173 secs. :x-pack:plugin:security:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:security:copyPluginPropertiesTemplate Task ':x-pack:plugin:security:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :x-pack:plugin:security:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :x-pack:plugin:security:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:security:pluginProperties Task ':x-pack:plugin:security:pluginProperties' is not up-to-date because: No history is available. :x-pack:plugin:security:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.016 secs. :x-pack:plugin:security:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:security:bundlePlugin Task ':x-pack:plugin:security:bundlePlugin' is not up-to-date because: No history is available. :x-pack:plugin:security:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.404 secs. :x-pack:plugin:sql:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:generateNotice Task ':x-pack:plugin:sql:generateNotice' is not up-to-date because: No history is available. :x-pack:plugin:sql:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.002 secs. :x-pack:plugin:sql:sql-proto:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:sql-proto:compileJava Task ':x-pack:plugin:sql:sql-proto:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':x-pack:plugin:sql:sql-proto:compileJava'. :x-pack:plugin:sql:sql-proto:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\sql\sql-proto Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\sql\sql-proto\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.01 secs. Created jar classpath snapshot for incremental compilation in 0.001 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :x-pack:plugin:sql:sql-proto:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.853 secs. :x-pack:plugin:sql:sql-proto:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:sql-proto:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\sql\sql-proto\src\main\resources', not found Skipping task ':x-pack:plugin:sql:sql-proto:processResources' as it has no source files and no previous output files. :x-pack:plugin:sql:sql-proto:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:sql:sql-proto:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:sql-proto:classes Skipping task ':x-pack:plugin:sql:sql-proto:classes' as it has no actions. :x-pack:plugin:sql:sql-proto:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:sql:sql-proto:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:sql-proto:jar Task ':x-pack:plugin:sql:sql-proto:jar' is not up-to-date because: No history is available. :x-pack:plugin:sql:sql-proto:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.019 secs. :x-pack:plugin:sql:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:compileJava Task ':x-pack:plugin:sql:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':x-pack:plugin:sql:compileJava'. :x-pack:plugin:sql:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\sql Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\sql\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Class dependency analysis for incremental compilation took 0.356 secs. Created jar classpath snapshot for incremental compilation in 0.011 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :x-pack:plugin:sql:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 8.275 secs. :x-pack:plugin:sql:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\sql\src\main\resources', not found Skipping task ':x-pack:plugin:sql:processResources' as it has no source files and no previous output files. :x-pack:plugin:sql:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:sql:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:classes Skipping task ':x-pack:plugin:sql:classes' as it has no actions. :x-pack:plugin:sql:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:sql:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:jar Task ':x-pack:plugin:sql:jar' is not up-to-date because: No history is available. :x-pack:plugin:sql:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.24 secs. :x-pack:plugin:sql:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:copyPluginPropertiesTemplate Task ':x-pack:plugin:sql:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :x-pack:plugin:sql:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :x-pack:plugin:sql:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:pluginProperties Task ':x-pack:plugin:sql:pluginProperties' is not up-to-date because: No history is available. :x-pack:plugin:sql:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.012 secs. :x-pack:plugin:sql:sql-shared-client:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:sql-shared-client:compileJava Task ':x-pack:plugin:sql:sql-shared-client:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':x-pack:plugin:sql:sql-shared-client:compileJava'. :x-pack:plugin:sql:sql-shared-client:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\sql\sql-shared-client Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\sql\sql-shared-client\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.009 secs. Created jar classpath snapshot for incremental compilation in 0.001 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :x-pack:plugin:sql:sql-shared-client:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.919 secs. :x-pack:plugin:sql:sql-shared-client:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:sql-shared-client:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\sql\sql-shared-client\src\main\resources', not found Skipping task ':x-pack:plugin:sql:sql-shared-client:processResources' as it has no source files and no previous output files. :x-pack:plugin:sql:sql-shared-client:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:sql:sql-shared-client:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:sql-shared-client:classes Skipping task ':x-pack:plugin:sql:sql-shared-client:classes' as it has no actions. :x-pack:plugin:sql:sql-shared-client:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:sql:sql-shared-client:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:sql-shared-client:jar Task ':x-pack:plugin:sql:sql-shared-client:jar' is not up-to-date because: No history is available. :x-pack:plugin:sql:sql-shared-client:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.025 secs. :x-pack:plugin:sql:sql-cli:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:sql-cli:compileJava Task ':x-pack:plugin:sql:sql-cli:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':x-pack:plugin:sql:sql-cli:compileJava'. :x-pack:plugin:sql:sql-cli:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\sql\sql-cli Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\sql\sql-cli\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Class dependency analysis for incremental compilation took 0.012 secs. Created jar classpath snapshot for incremental compilation in 0.084 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :x-pack:plugin:sql:sql-cli:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.69 secs. :x-pack:plugin:sql:sql-cli:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:sql-cli:processResources Task ':x-pack:plugin:sql:sql-cli:processResources' is not up-to-date because: No history is available. :x-pack:plugin:sql:sql-cli:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.003 secs. :x-pack:plugin:sql:sql-cli:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:sql-cli:classes Skipping task ':x-pack:plugin:sql:sql-cli:classes' as it has no actions. :x-pack:plugin:sql:sql-cli:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:sql:sql-cli:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:sql-cli:jar Task ':x-pack:plugin:sql:sql-cli:jar' is not up-to-date because: No history is available. :x-pack:plugin:sql:sql-cli:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.591 secs. :x-pack:plugin:sql:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:sql:bundlePlugin Task ':x-pack:plugin:sql:bundlePlugin' is not up-to-date because: No history is available. :x-pack:plugin:sql:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.889 secs. :x-pack:plugin:upgrade:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:upgrade:generateNotice Task ':x-pack:plugin:upgrade:generateNotice' is not up-to-date because: No history is available. :x-pack:plugin:upgrade:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :x-pack:plugin:upgrade:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:upgrade:compileJava Task ':x-pack:plugin:upgrade:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':x-pack:plugin:upgrade:compileJava'. :x-pack:plugin:upgrade:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\upgrade Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\upgrade\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\upgrade\src\main\java\org\elasticsearch\xpack\upgrade\InternalIndexReindexer.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\upgrade\src\main\java\org\elasticsearch\xpack\upgrade\IndexUpgradeService.java uses unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. Class dependency analysis for incremental compilation took 0.008 secs. Created jar classpath snapshot for incremental compilation in 0.005 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :x-pack:plugin:upgrade:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.951 secs. :x-pack:plugin:upgrade:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:upgrade:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\upgrade\src\main\resources', not found Skipping task ':x-pack:plugin:upgrade:processResources' as it has no source files and no previous output files. :x-pack:plugin:upgrade:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:upgrade:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:upgrade:classes Skipping task ':x-pack:plugin:upgrade:classes' as it has no actions. :x-pack:plugin:upgrade:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:upgrade:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:upgrade:jar Task ':x-pack:plugin:upgrade:jar' is not up-to-date because: No history is available. :x-pack:plugin:upgrade:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.015 secs. :x-pack:plugin:upgrade:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:upgrade:copyPluginPropertiesTemplate Task ':x-pack:plugin:upgrade:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :x-pack:plugin:upgrade:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :x-pack:plugin:upgrade:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:upgrade:pluginProperties Task ':x-pack:plugin:upgrade:pluginProperties' is not up-to-date because: No history is available. :x-pack:plugin:upgrade:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.014 secs. :x-pack:plugin:upgrade:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:upgrade:bundlePlugin Task ':x-pack:plugin:upgrade:bundlePlugin' is not up-to-date because: No history is available. :x-pack:plugin:upgrade:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.009 secs. :x-pack:plugin:watcher:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:watcher:generateNotice Task ':x-pack:plugin:watcher:generateNotice' is not up-to-date because: No history is available. :x-pack:plugin:watcher:generateNotice (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.003 secs. :x-pack:plugin:watcher:compileJava (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:watcher:compileJava Task ':x-pack:plugin:watcher:compileJava' is not up-to-date because: No history is available. All input files are considered out-of-date for incremental task ':x-pack:plugin:watcher:compileJava'. :x-pack:plugin:watcher:compileJava - is not incremental (e.g. outputs have changed, no previous execution, etc.). Compiling with Java command line compiler 'C:\Users\jenkins\.java\java10\bin\javac.exe'. Starting process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\watcher Command: C:\Users\jenkins\.java\java10\bin\javac.exe -J-Xmx512m @C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\watcher\build\tmp\compileJava\java-compiler-args.txt Successfully started process 'command 'C:\Users\jenkins\.java\java10\bin\javac.exe'' Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. Class dependency analysis for incremental compilation took 0.182 secs. Created jar classpath snapshot for incremental compilation in 0.346 secs. Written jar classpath snapshot for incremental compilation in 0.0 secs. :x-pack:plugin:watcher:compileJava (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 6.794 secs. :x-pack:plugin:watcher:processResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:watcher:processResources NO-SOURCE file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\x-pack\plugin\watcher\src\main\resources', not found Skipping task ':x-pack:plugin:watcher:processResources' as it has no source files and no previous output files. :x-pack:plugin:watcher:processResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:watcher:classes (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:watcher:classes Skipping task ':x-pack:plugin:watcher:classes' as it has no actions. :x-pack:plugin:watcher:classes (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :x-pack:plugin:watcher:jar (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:watcher:jar Task ':x-pack:plugin:watcher:jar' is not up-to-date because: No history is available. :x-pack:plugin:watcher:jar (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.177 secs. :x-pack:plugin:watcher:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:watcher:copyPluginPropertiesTemplate Task ':x-pack:plugin:watcher:copyPluginPropertiesTemplate' is not up-to-date because: Task has not declared any outputs despite executing actions. :x-pack:plugin:watcher:copyPluginPropertiesTemplate (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :x-pack:plugin:watcher:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:watcher:pluginProperties Task ':x-pack:plugin:watcher:pluginProperties' is not up-to-date because: No history is available. :x-pack:plugin:watcher:pluginProperties (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.012 secs. :x-pack:plugin:watcher:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:watcher:bundlePlugin Task ':x-pack:plugin:watcher:bundlePlugin' is not up-to-date because: No history is available. :x-pack:plugin:watcher:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.193 secs. :x-pack:plugin:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :x-pack:plugin:bundlePlugin Task ':x-pack:plugin:bundlePlugin' is not up-to-date because: No history is available. :x-pack:plugin:bundlePlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 7.015 secs. :distribution:processDefaultOutputs (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:processDefaultOutputs Task ':distribution:processDefaultOutputs' is not up-to-date because: No history is available. :distribution:processDefaultOutputs (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.834 secs. :distribution:buildDefaultBin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:buildDefaultBin Skipping task ':distribution:buildDefaultBin' as it has no actions. :distribution:buildDefaultBin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :distribution:buildDefaultConfig (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:buildDefaultConfig Skipping task ':distribution:buildDefaultConfig' as it has no actions. :distribution:buildDefaultConfig (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :distribution:buildDefaultLog4jConfig (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:buildDefaultLog4jConfig Task ':distribution:buildDefaultLog4jConfig' is not up-to-date because: No history is available. :distribution:buildDefaultLog4jConfig (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.016 secs. :distribution:processTransportOutputs (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:processTransportOutputs Task ':distribution:processTransportOutputs' is not up-to-date because: No history is available. :distribution:processTransportOutputs (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.058 secs. :distribution:buildTransportModules (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:buildTransportModules Skipping task ':distribution:buildTransportModules' as it has no actions. :distribution:buildTransportModules (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :distribution:archives:buildIntegTestZip (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :distribution:archives:buildIntegTestZip Task ':distribution:archives:buildIntegTestZip' is not up-to-date because: No history is available. :distribution:archives:buildIntegTestZip (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 2.094 secs. :plugins:analysis-icu:integTestCluster#prepareCluster.cleanShared (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:integTestCluster#prepareCluster.cleanShared Task ':plugins:analysis-icu:integTestCluster#prepareCluster.cleanShared' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:analysis-icu:integTestCluster#prepareCluster.cleanShared (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-icu:integTestCluster#clean (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:integTestCluster#clean UP-TO-DATE Task ':plugins:analysis-icu:integTestCluster#clean' is not up-to-date because: Task has not declared any outputs despite executing actions. :plugins:analysis-icu:integTestCluster#clean (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-icu:integTestCluster#createCwd (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:integTestCluster#createCwd Task ':plugins:analysis-icu:integTestCluster#createCwd' is not up-to-date because: No history is available. :plugins:analysis-icu:integTestCluster#createCwd (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:analysis-icu:integTestCluster#checkPrevious (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:integTestCluster#checkPrevious SKIPPED Skipping task ':plugins:analysis-icu:integTestCluster#checkPrevious' as task onlyIf is false. :plugins:analysis-icu:integTestCluster#checkPrevious (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-icu:integTestCluster#stopPrevious (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:integTestCluster#stopPrevious SKIPPED Skipping task ':plugins:analysis-icu:integTestCluster#stopPrevious' as task onlyIf is false. :plugins:analysis-icu:integTestCluster#stopPrevious (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.0 secs. :plugins:analysis-icu:integTestCluster#extract (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:integTestCluster#extract Task ':plugins:analysis-icu:integTestCluster#extract' is not up-to-date because: No history is available. :plugins:analysis-icu:integTestCluster#extract (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.793 secs. :plugins:analysis-icu:integTestCluster#configure (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:integTestCluster#configure Task ':plugins:analysis-icu:integTestCluster#configure' is not up-to-date because: Task has not declared any outputs despite executing actions. Configuring C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-icu\build\cluster\integTestCluster node0\elasticsearch-7.0.0-alpha1-SNAPSHOT\config\elasticsearch.yml :plugins:analysis-icu:integTestCluster#configure (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.001 secs. :plugins:analysis-icu:integTestCluster#copyPlugins (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:integTestCluster#copyPlugins Task ':plugins:analysis-icu:integTestCluster#copyPlugins' is not up-to-date because: No history is available. :plugins:analysis-icu:integTestCluster#copyPlugins (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.066 secs. :plugins:analysis-icu:integTestCluster#installAnalysisIcuPlugin (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:integTestCluster#installAnalysisIcuPlugin Task ':plugins:analysis-icu:integTestCluster#installAnalysisIcuPlugin' is not up-to-date because: Task has not declared any outputs despite executing actions. Starting process 'command 'cmd''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-icu\build\cluster\integTestCluster node0\cwd Command: cmd /C call C:\Users\jenkins\WORKSP~1\ELASTI~1\plugins\ANALYS~1\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch-plugin install --batch file:/C:/Users/jenkins/workspace/elastic+elasticsearch+master+multijob-windows-compatibility/plugins/analysis-icu/build/cluster/integTestCluster%20node0/plugins%20tmp/analysis-icu-7.0.0-alpha1-SNAPSHOT.zip Successfully started process 'command 'cmd'' -> Downloading file:/C:/Users/jenkins/workspace/elastic elasticsearch master multijob-windows-compatibility/plugins/analysis-icu/build/cluster/integTestCluster node0/plugins tmp/analysis-icu-7.0.0-alpha1-SNAPSHOT.zip [ ] 1%?? [> ] 2%?? [> ] 3%?? [=> ] 4%?? [=> ] 5%?? [==> ] 6%?? [==> ] 7%?? [===> ] 8%?? [===> ] 9%?? [====> ] 10%?? [====> ] 11%?? [=====> ] 12%?? [=====> ] 13%?? [======> ] 14%?? [======> ] 15%?? [=======> ] 16%?? [=======> ] 17%?? [========> ] 18%?? [========> ] 19%?? [=========> ] 20%?? [=========> ] 21%?? [==========> ] 22%?? [==========> ] 23%?? [===========> ] 24%?? [===========> ] 25%?? [============> ] 26%?? [============> ] 27%?? [=============> ] 28%?? [=============> ] 29%?? [==============> ] 30%?? [==============> ] 31%?? [===============> ] 32%?? [===============> ] 33%?? [================> ] 34%?? [================> ] 35%?? [=================> ] 36%?? [=================> ] 37%?? [==================> ] 38%?? [==================> ] 39%?? [===================> ] 40%?? [===================> ] 41%?? [====================> ] 42%?? [====================> ] 43%?? [=====================> ] 44%?? [=====================> ] 45%?? [======================> ] 46%?? [======================> ] 47%?? [=======================> ] 48%?? [=======================> ] 49%?? [========================> ] 50%?? [========================> ] 51%?? [=========================> ] 52%?? [=========================> ] 53%?? [==========================> ] 54%?? [==========================> ] 55%?? [===========================> ] 56%?? [===========================> ] 57%?? [============================> ] 58%?? [============================> ] 59%?? [=============================> ] 60%?? [=============================> ] 61%?? [==============================> ] 62%?? [==============================> ] 63%?? [===============================> ] 64%?? [===============================> ] 65%?? [================================> ] 66%?? [================================> ] 67%?? [=================================> ] 68%?? [=================================> ] 69%?? [==================================> ] 70%?? [==================================> ] 71%?? [===================================> ] 72%?? [===================================> ] 73%?? [====================================> ] 74%?? [====================================> ] 75%?? [=====================================> ] 76%?? [=====================================> ] 77%?? [======================================> ] 78%?? [======================================> ] 79%?? [=======================================> ] 80%?? [=======================================> ] 81%?? [========================================> ] 82%?? [========================================> ] 83%?? [=========================================> ] 84%?? [=========================================> ] 85%?? [==========================================> ] 86%?? [==========================================> ] 87%?? [===========================================> ] 88%?? [===========================================> ] 89%?? [============================================> ] 90%?? [============================================> ] 91%?? [=============================================> ] 92%?? [=============================================> ] 93%?? [==============================================> ] 94%?? [==============================================> ] 95%?? [===============================================> ] 96%?? [===============================================> ] 97%?? [================================================>] 98%?? [================================================>] 99%?? [=================================================] 100%?? -> Installed analysis-icu :plugins:analysis-icu:integTestCluster#installAnalysisIcuPlugin (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.369 secs. :plugins:analysis-icu:integTestCluster#start (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:integTestCluster#start Task ':plugins:analysis-icu:integTestCluster#start' is not up-to-date because: Task has not declared any outputs despite executing actions. Node 0 configuration: |----------------------------------------- | cwd: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-icu\build\cluster\integTestCluster node0\cwd | command: cmd /C " C:\Users\jenkins\WORKSP~1\ELASTI~1\plugins\ANALYS~1\build\cluster\INTEGT~1\cwd\run.bat -E node.portsfile=true -E path.data=C:\Users\jenkins\WORKSP~1\ELASTI~1\plugins\ANALYS~1\build\cluster\INTEGT~1\ELASTI~1.0-A\data " | environment: | JAVA_HOME: C:\Users\jenkins\.java\java8 | ES_JAVA_OPTS: -ea -esa -Xms512m -Xmx512m | ES_PATH_CONF: C:\Users\jenkins\WORKSP~1\ELASTI~1\plugins\ANALYS~1\build\cluster\INTEGT~1\ELASTI~1.0-A\config | | [run.bat] "C:\Users\jenkins\WORKSP~1\ELASTI~1\plugins\ANALYS~1\build\cluster\INTEGT~1\ELASTI~1.0-A\bin\elasticsearch.bat" %* > run.log 2>&1 if "%errorlevel%" neq "0" ( type nul >> run.failed ) | | [elasticsearch.yml] | cluster.name: plugins_analysis-icu_integTestCluster | node.name: node-0 | pidfile: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-icu\build\cluster\integTestCluster node0\es.pid | path.repo: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-icu\build\cluster\shared/repo | path.shared_data: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-icu\build\cluster\shared/ | node.attr.testattr: test | node.max_local_storage_nodes: 1 | http.port: 0 | transport.tcp.port: 0 | cluster.routing.allocation.disk.watermark.low: 1b | cluster.routing.allocation.disk.watermark.high: 1b | cluster.routing.allocation.disk.watermark.flood_stage: 1b | script.max_compilations_rate: 2048/1m |----------------------------------------- :plugins:analysis-icu:integTestCluster#start (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 1.023 secs. :plugins:analysis-icu:processTestResources (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:processTestResources file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-icu\src\main\plugin-metadata', not found Task ':plugins:analysis-icu:processTestResources' is not up-to-date because: No history is available. file or directory 'C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-icu\src\main\plugin-metadata', not found :plugins:analysis-icu:processTestResources (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.017 secs. :plugins:analysis-icu:copyRestSpec (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:copyRestSpec Task ':plugins:analysis-icu:copyRestSpec' is not up-to-date because: No history is available. :plugins:analysis-icu:copyRestSpec (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.116 secs. :server:integTest (Thread[Task worker for ':' Thread 3,5,main]) started. Started J1 PID(2948@SLAVE-0633DD87F). Started J0 PID(6640@SLAVE-0633DD87F). Started J2 PID(9936@SLAVE-0633DD87F). > Task :server:integTest Task ':server:integTest' is not up-to-date because: Task.upToDateWhen is false. [ant:junit4] says ahoj! Master seed: 734B52BB31887C24 ==> Test Info: seed=734B52BB31887C24; jvms=3; suites=296 HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:37:02, stalled for 10.6s at: SearchFieldsIT.testSearchFieldsMetaData HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:37:02, stalled for 10.2s at: IndexRequestBuilderIT.testOddNumberOfSourceObjects HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:37:04, stalled for 11.0s at: HighlighterSearchIT.testHighlighterWithWordBoundaryScanner Suite: org.elasticsearch.index.IndexRequestBuilderIT Completed [1/296] on J2 in 24.12s, 2 tests Suite: org.elasticsearch.index.search.MatchPhraseQueryIT Completed [2/296] on J2 in 1.79s, 1 test Suite: org.elasticsearch.search.fields.SearchFieldsIT Completed [3/296] on J1 in 30.80s, 14 tests Suite: org.elasticsearch.similarity.SimilarityIT Completed [4/296] on J1 in 1.01s, 1 test Suite: org.elasticsearch.search.geo.GeoDistanceIT Completed [5/296] on J1 in 4.82s, 2 tests Suite: org.elasticsearch.index.fielddata.FieldDataLoadingIT Completed [6/296] on J1 in 3.12s, 1 test Suite: org.elasticsearch.index.store.CorruptedTranslogIT Completed [7/296] on J1 in 2.14s, 1 test Suite: org.elasticsearch.broadcast.BroadcastActionsIT Completed [8/296] on J1 in 1.52s, 1 test HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:37:32, stalled for 12.0s at: DiscoveryDisruptionIT.testElectMasterWithLatestVersion HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:37:42, stalled for 22.0s at: DiscoveryDisruptionIT.testElectMasterWithLatestVersion HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:37:48, stalled for 11.5s at: ClusterDisruptionIT.testIndicesDeleted Suite: org.elasticsearch.search.fetch.subphase.highlight.HighlighterSearchIT Completed [9/296] on J0 in 73.71s, 83 tests HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:38:12, stalled for 11.8s at: DiscoveryDisruptionIT.testIsolatedUnicastNodes Suite: org.elasticsearch.search.suggest.SuggestSearchIT Completed [10/296] on J0 in 17.78s, 18 tests Suite: org.elasticsearch.discovery.DiscoveryDisruptionIT Completed [11/296] on J2 in 72.02s, 6 tests Suite: org.elasticsearch.indices.analysis.PreBuiltAnalyzerIntegrationIT Completed [12/296] on J2 in 2.53s, 1 test Suite: org.elasticsearch.discovery.ClusterDisruptionIT Completed [13/296] on J1 in 60.42s, 6 tests Suite: org.elasticsearch.snapshots.MetadataLoadingDuringSnapshotRestoreIT Completed [14/296] on J2 in 4.92s, 1 test Suite: org.elasticsearch.search.geo.GeoBoundingBoxIT Completed [15/296] on J0 in 13.79s, 3 tests Suite: org.elasticsearch.action.termvectors.MultiTermVectorsIT Completed [16/296] on J1 in 5.22s, 3 tests Suite: org.elasticsearch.search.query.ExistsIT Completed [17/296] on J2 in 4.26s, 2 tests Suite: org.elasticsearch.index.engine.InternalEngineMergeIT Completed [18/296] on J0 in 8.08s, 1 test Suite: org.elasticsearch.client.node.NodeClientIT Completed [19/296] on J2 in 9.32s, 1 test HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:38:46, stalled for 11.8s at: SimpleNestedIT.testSimpleNestedSorting Suite: org.elasticsearch.search.nested.SimpleNestedIT Completed [20/296] on J1 in 20.40s, 10 tests Suite: org.elasticsearch.index.store.CorruptedFileIT Completed [21/296] on J0 in 18.10s, 6 tests Suite: org.elasticsearch.action.admin.indices.rollover.RolloverIT Completed [22/296] on J2 in 14.79s, 10 tests Suite: org.elasticsearch.blocks.SimpleBlocksIT Completed [23/296] on J0 in 3.17s, 2 tests Suite: org.elasticsearch.cluster.shards.ClusterSearchShardsIT Completed [24/296] on J0 in 2.94s, 4 tests Suite: org.elasticsearch.action.termvectors.GetTermVectorsIT Completed [25/296] on J0 in 9.94s, 19 tests Suite: org.elasticsearch.routing.SimpleRoutingIT Completed [26/296] on J2 in 14.44s, 5 tests Suite: org.elasticsearch.search.msearch.MultiSearchIT Completed [27/296] on J0 in 2.59s, 2 tests Suite: org.elasticsearch.indices.state.RareClusterStateIT Completed [28/296] on J1 in 24.43s, 4 tests Suite: org.elasticsearch.search.basic.SearchRedStateIndexIT Completed [29/296] on J1 in 4.50s, 4 tests Suite: org.elasticsearch.cluster.SimpleClusterStateIT Completed [30/296] on J2 in 14.33s, 10 tests Suite: org.elasticsearch.transport.ActionNamesIT Completed [31/296] on J2 in 0.78s, 1 test Suite: org.elasticsearch.action.ListenerActionIT Completed [32/296] on J1 in 8.57s, 1 test Suite: org.elasticsearch.indices.state.OpenCloseIndexIT Completed [33/296] on J0 in 19.90s, 22 tests Suite: org.elasticsearch.search.aggregations.metrics.ScriptedMetricIT Completed [34/296] on J1 in 4.94s, 17 tests Suite: org.elasticsearch.search.aggregations.pipeline.serialdiff.SerialDiffIT Completed [35/296] on J0 in 3.51s, 2 tests Suite: org.elasticsearch.search.aggregations.bucket.AdjacencyMatrixIT Completed [36/296] on J2 in 13.60s, 8 tests HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:39:55, stalled for 11.9s at: RandomScoreFunctionIT.testConsistentHitsWithSameSeed Suite: org.elasticsearch.snapshots.FsBlobStoreRepositoryIT Completed [37/296] on J1 in 18.87s, 3 tests Suite: org.elasticsearch.search.slice.SearchSliceIT Completed [38/296] on J1 in 6.67s, 4 tests HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:40:05, stalled for 21.9s at: RandomScoreFunctionIT.testConsistentHitsWithSameSeed Suite: org.elasticsearch.indices.mapping.ConcurrentDynamicTemplateIT Completed [39/296] on J1 in 6.74s, 1 test Suite: org.elasticsearch.search.functionscore.RandomScoreFunctionIT Completed [40/296] on J2 in 36.34s, 6 tests Suite: org.elasticsearch.search.functionscore.QueryRescorerIT Completed [41/296] on J1 in 17.01s, 11 tests Suite: org.elasticsearch.action.IndicesRequestIT Completed [42/296] on J0 in 48.43s, 31 tests Suite: org.elasticsearch.search.aggregations.metrics.MinIT Completed [43/296] on J0 in 7.33s, 18 tests HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:40:45, stalled for 10.2s at: RecoveryWhileUnderLoadIT.testRecoverWhileUnderLoadWithReducedAllowedNodes Suite: org.elasticsearch.gateway.GatewayIndexStateIT Completed [44/296] on J2 in 26.80s, 10 tests Suite: org.elasticsearch.search.aggregations.bucket.DoubleTermsIT Completed [45/296] on J2 in 7.76s, 30 tests HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:40:55, stalled for 11.7s at: DedicatedClusterSnapshotRestoreIT.testMasterAndDataShutdownDuringSnapshot Suite: org.elasticsearch.cluster.UpdateSettingsValidationIT Completed [46/296] on J2 in 2.01s, 1 test Suite: org.elasticsearch.recovery.RecoveryWhileUnderLoadIT Completed [47/296] on J1 in 29.17s, 4 tests Suite: org.elasticsearch.search.aggregations.bucket.DateHistogramOffsetIT Completed [48/296] on J2 in 2.17s, 3 tests Suite: org.elasticsearch.indices.IndicesRequestCacheIT Completed [49/296] on J1 in 5.53s, 7 tests Suite: org.elasticsearch.search.aggregations.metrics.GeoCentroidIT Completed [50/296] on J2 in 4.23s, 7 tests HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:41:05, stalled for 21.9s at: DedicatedClusterSnapshotRestoreIT.testMasterAndDataShutdownDuringSnapshot HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:41:13, stalled for 11.1s at: ClusterStateDiffIT.testClusterStateDiffSerialization HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:41:13, stalled for 11.6s at: SplitIndexIT.testSplitIndexPrimaryTerm Suite: org.elasticsearch.cluster.ClusterStateDiffIT Completed [51/296] on J2 in 20.92s, 1 test HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:41:23, stalled for 21.7s at: SplitIndexIT.testSplitIndexPrimaryTerm HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:41:33, stalled for 31.7s at: SplitIndexIT.testSplitIndexPrimaryTerm Suite: org.elasticsearch.snapshots.DedicatedClusterSnapshotRestoreIT IGNOR/A 0.01s J0 | DedicatedClusterSnapshotRestoreIT.testMasterShutdownDuringFailedSnapshot > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/25281)) Completed [52/296] on J0 in 58.74s, 13 tests, 1 skipped Suite: org.elasticsearch.action.admin.indices.create.SplitIndexIT 1> [2018-05-05T20:41:01,898][INFO ][o.e.a.a.i.c.SplitIndexIT ] [testSplitIndexPrimaryTerm]: before test 1> [2018-05-05T20:41:01,899][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testSplitIndexPrimaryTerm]: setting up test 1> [2018-05-05T20:41:01,900][INFO ][o.e.t.InternalTestCluster] Setup InternalTestCluster [SUITE-CHILD_VM=[1]-CLUSTER_SEED=[7796208738573032173]-HASH=[56C358854572DC]-cluster] with seed [6C31B28AA9711AED] using [0] dedicated masters, [2] (data) nodes and [0] coord only nodes (min_master_nodes are [auto-managed]) 1> [2018-05-05T20:41:01,902][INFO ][o.e.n.Node ] [node_s0] initializing ... 1> [2018-05-05T20:41:01,909][INFO ][o.e.e.NodeEnvironment ] [node_s0] using [1] data paths, mounts [[Windows (C:)]], net usable_space [379.2gb], net total_space [499.9gb], types [NTFS] 1> [2018-05-05T20:41:01,910][INFO ][o.e.e.NodeEnvironment ] [node_s0] heap size [491mb], compressed ordinary object pointers [true] 1> [2018-05-05T20:41:01,911][INFO ][o.e.n.Node ] [node_s0] node name [node_s0], node ID [YOWAmP1SSD2IFm9IQ_0seA] 1> [2018-05-05T20:41:01,911][INFO ][o.e.n.Node ] [node_s0] version[7.0.0-alpha1-SNAPSHOT], pid[2948], build[unknown/unknown/Unknown/Unknown], OS[Windows Server 2016/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_162/25.162-b12] 1> [2018-05-05T20:41:01,911][INFO ][o.e.n.Node ] [node_s0] JVM arguments [-ea, -esa, -Xmx512m, -Xms512m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\heapdump, -Dtests.prefix=tests, -Dtests.seed=734B52BB31887C24, -Dtests.security.manager=true, -Djava.io.tmpdir=./temp, -Dtests.gradle=true, -Dtests.artifact=server, -Djava.awt.headless=true, -Dtests.task=:server:integTest, -Djna.nosys=true, -Dtests.logger.level=WARN, -Djunit4.childvm.cwd=C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1, -Djunit4.childvm.id=1, -Djunit4.childvm.count=3] 1> [2018-05-05T20:41:01,911][WARN ][o.e.n.Node ] [node_s0] version [7.0.0-alpha1-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production 1> [2018-05-05T20:41:01,913][INFO ][o.e.p.PluginsService ] [node_s0] no modules loaded 1> [2018-05-05T20:41:01,913][INFO ][o.e.p.PluginsService ] [node_s0] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] 1> [2018-05-05T20:41:01,913][INFO ][o.e.p.PluginsService ] [node_s0] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] 1> [2018-05-05T20:41:01,913][INFO ][o.e.p.PluginsService ] [node_s0] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] 1> [2018-05-05T20:41:01,913][INFO ][o.e.p.PluginsService ] [node_s0] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin] 1> [2018-05-05T20:41:01,913][INFO ][o.e.p.PluginsService ] [node_s0] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin] 1> [2018-05-05T20:41:01,930][INFO ][o.e.d.DiscoveryModule ] [node_s0] using discovery type [test-zen] 1> [2018-05-05T20:41:01,942][INFO ][o.e.n.Node ] [node_s0] initialized 1> [2018-05-05T20:41:01,942][INFO ][o.e.n.Node ] [node_s1] initializing ... 1> [2018-05-05T20:41:01,951][INFO ][o.e.e.NodeEnvironment ] [node_s1] using [1] data paths, mounts [[Windows (C:)]], net usable_space [379.2gb], net total_space [499.9gb], types [NTFS] 1> [2018-05-05T20:41:01,951][INFO ][o.e.e.NodeEnvironment ] [node_s1] heap size [491mb], compressed ordinary object pointers [true] 1> [2018-05-05T20:41:01,953][INFO ][o.e.n.Node ] [node_s1] node name [node_s1], node ID [mIq8KinYToaJSuyTqY1_hQ] 1> [2018-05-05T20:41:01,953][INFO ][o.e.n.Node ] [node_s1] version[7.0.0-alpha1-SNAPSHOT], pid[2948], build[unknown/unknown/Unknown/Unknown], OS[Windows Server 2016/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_162/25.162-b12] 1> [2018-05-05T20:41:01,953][INFO ][o.e.n.Node ] [node_s1] JVM arguments [-ea, -esa, -Xmx512m, -Xms512m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\heapdump, -Dtests.prefix=tests, -Dtests.seed=734B52BB31887C24, -Dtests.security.manager=true, -Djava.io.tmpdir=./temp, -Dtests.gradle=true, -Dtests.artifact=server, -Djava.awt.headless=true, -Dtests.task=:server:integTest, -Djna.nosys=true, -Dtests.logger.level=WARN, -Djunit4.childvm.cwd=C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1, -Djunit4.childvm.id=1, -Djunit4.childvm.count=3] 1> [2018-05-05T20:41:01,953][WARN ][o.e.n.Node ] [node_s1] version [7.0.0-alpha1-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production 1> [2018-05-05T20:41:01,954][INFO ][o.e.p.PluginsService ] [node_s1] no modules loaded 1> [2018-05-05T20:41:01,954][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] 1> [2018-05-05T20:41:01,954][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] 1> [2018-05-05T20:41:01,954][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] 1> [2018-05-05T20:41:01,954][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin] 1> [2018-05-05T20:41:01,954][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin] 1> [2018-05-05T20:41:01,975][INFO ][o.e.d.DiscoveryModule ] [node_s1] using discovery type [test-zen] 1> [2018-05-05T20:41:01,987][INFO ][o.e.n.Node ] [node_s1] initialized 1> [2018-05-05T20:41:01,987][INFO ][o.e.n.Node ] [node_s1] starting ... 1> [2018-05-05T20:41:01,987][INFO ][o.e.n.Node ] [node_s0] starting ... 1> [2018-05-05T20:41:02,032][INFO ][o.e.t.TransportService ] [node_s0] publish_address {127.0.0.1:54139}, bound_addresses {127.0.0.1:54139}, {[::1]:54140} 1> [2018-05-05T20:41:02,033][INFO ][o.e.t.d.MockZenPing ] [node_s0] pinging using mock zen ping 1> [2018-05-05T20:41:02,033][WARN ][o.e.t.d.TestZenDiscovery ] [node_s0] not enough master nodes discovered during pinging (found [[Candidate{node={node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}, clusterStateVersion=-1}]], but needed [2]), pinging again 1> [2018-05-05T20:41:02,033][INFO ][o.e.t.d.MockZenPing ] [node_s0] pinging using mock zen ping 1> [2018-05-05T20:41:02,035][INFO ][o.e.t.TransportService ] [node_s1] publish_address {127.0.0.1:54141}, bound_addresses {127.0.0.1:54141}, {[::1]:54142} 1> [2018-05-05T20:41:02,036][INFO ][o.e.t.d.MockZenPing ] [node_s1] pinging using mock zen ping 1> [2018-05-05T20:41:02,040][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-elected-as-master ([1] nodes joined)[, ], reason: master node changed {previous [], current [{node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]}, added {{node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141},} 1> [2018-05-05T20:41:02,041][INFO ][o.e.c.s.ClusterApplierService] [node_s1] master node changed {previous [], current [{node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]}, added {{node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139},}, reason: apply cluster state (from master [master {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} committed version [1]]) 1> [2018-05-05T20:41:02,041][INFO ][o.e.n.Node ] [node_s1] started 1> [2018-05-05T20:41:02,042][INFO ][o.e.c.s.ClusterApplierService] [node_s0] master node changed {previous [], current [{node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]}, added {{node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141},}, reason: apply cluster state (from master [master {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} committed version [1] source [zen-disco-elected-as-master ([1] nodes joined)[, ]]]) 1> [2018-05-05T20:41:02,042][INFO ][o.e.n.Node ] [node_s0] started 1> [2018-05-05T20:41:02,043][INFO ][o.e.p.PluginsService ] [transport_client_node_s1] no modules loaded 1> [2018-05-05T20:41:02,043][INFO ][o.e.p.PluginsService ] [transport_client_node_s1] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin] 1> [2018-05-05T20:41:02,054][INFO ][o.e.g.GatewayService ] [node_s0] recovered [0] indices into cluster_state 1> [2018-05-05T20:41:02,057][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [random_index_template] for index patterns [*] 1> [2018-05-05T20:41:02,065][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testSplitIndexPrimaryTerm]: all set up test 1> [2018-05-05T20:41:02,068][INFO ][o.e.a.a.i.c.SplitIndexIT ] using custom data_path for index: [xVyPCIeMVk] 1> [2018-05-05T20:41:02,069][INFO ][o.e.p.PluginsService ] [transport_client_node_s0] no modules loaded 1> [2018-05-05T20:41:02,069][INFO ][o.e.p.PluginsService ] [transport_client_node_s0] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin] 1> [2018-05-05T20:41:02,077][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [source] creating index, cause [api], templates [random_index_template], shards [8]/[0], mappings [] 1> [2018-05-05T20:41:02,215][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][4], [source][2], [source][6], [source][0]] ...]). 1> [2018-05-05T20:41:02,237][WARN ][o.e.i.e.Engine ] [node_s0] [source][2] failed engine [test] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) [test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:02,240][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[source][2]] marking and sending shard failed due to [shard failure, reason [test]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:02,242][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [source/vHlv_SWpSoWaq6CvE9K2fw] create_mapping [type] 1> [2018-05-05T20:41:02,251][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [source][2], node[YOWAmP1SSD2IFm9IQ_0seA], [P], s[STARTED], a[id=LH1LLu7LSbCV9QTEFcX8ig], message [shard failure, reason [test]], failure [Exception[test]], markAsStale [true]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:02,253][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [RED] (reason: [shards failed [[source][2], [source][2]] ...]). 1> [2018-05-05T20:41:02,299][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[source][2]] ...]). 1> [2018-05-05T20:41:02,317][WARN ][o.e.i.e.Engine ] [node_s0] [source][4] failed engine [test] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) [test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:02,319][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[source][4]] marking and sending shard failed due to [shard failure, reason [test]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:02,321][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [source][4], node[YOWAmP1SSD2IFm9IQ_0seA], [P], s[STARTED], a[id=MLM_EdsjSAeUumNrTzmjbA], message [shard failure, reason [test]], failure [Exception[test]], markAsStale [true]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:02,323][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [RED] (reason: [shards failed [[source][4]] ...]). 1> [2018-05-05T20:41:02,423][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[source][4]] ...]). 1> [2018-05-05T20:41:02,442][WARN ][o.e.i.e.Engine ] [node_s1] [source][7] failed engine [test] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) [test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:02,445][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[source][7]] marking and sending shard failed due to [shard failure, reason [test]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:02,447][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [source][7], node[mIq8KinYToaJSuyTqY1_hQ], [P], s[STARTED], a[id=0-7qHNzFQs-5DGYU8sP8oA], message [shard failure, reason [test]], failure [NotSerializableExceptionWrapper[exception: test]], markAsStale [true]] 1> org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:02,450][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [RED] (reason: [shards failed [[source][7]] ...]). 1> [2018-05-05T20:41:02,697][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[source][7]] ...]). 1> [2018-05-05T20:41:02,725][WARN ][o.e.i.e.Engine ] [node_s0] [source][4] failed engine [test] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) [test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:02,728][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[source][4]] marking and sending shard failed due to [shard failure, reason [test]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:02,761][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [source][4], node[YOWAmP1SSD2IFm9IQ_0seA], [P], s[STARTED], a[id=MLM_EdsjSAeUumNrTzmjbA], message [shard failure, reason [test]], failure [Exception[test]], markAsStale [true]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:02,763][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [RED] (reason: [shards failed [[source][4]] ...]). 1> [2018-05-05T20:41:02,893][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[source][4]] ...]). 1> [2018-05-05T20:41:02,928][WARN ][o.e.i.e.Engine ] [node_s0] [source][6] failed engine [test] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) [test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:03,056][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[source][6]] marking and sending shard failed due to [shard failure, reason [test]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:03,058][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [source][6], node[YOWAmP1SSD2IFm9IQ_0seA], [P], s[STARTED], a[id=z34owuW4TS2RQMJxeRUlJg], message [shard failure, reason [test]], failure [Exception[test]], markAsStale [true]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:03,060][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [RED] (reason: [shards failed [[source][6]] ...]). 1> [2018-05-05T20:41:03,161][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[source][6]] ...]). 1> [2018-05-05T20:41:03,263][WARN ][o.e.i.e.Engine ] [node_s0] [source][2] failed engine [test] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) [test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:03,334][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[source][2]] marking and sending shard failed due to [shard failure, reason [test]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:03,406][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [source][2], node[YOWAmP1SSD2IFm9IQ_0seA], [P], s[STARTED], a[id=LH1LLu7LSbCV9QTEFcX8ig], message [shard failure, reason [test]], failure [Exception[test]], markAsStale [true]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:03,408][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [RED] (reason: [shards failed [[source][2], [source][2]] ...]). 1> [2018-05-05T20:41:03,457][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[source][2]] ...]). 1> [2018-05-05T20:41:03,623][WARN ][o.e.i.e.Engine ] [node_s0] [source][6] failed engine [test] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) [test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:03,625][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[source][6]] marking and sending shard failed due to [shard failure, reason [test]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:03,627][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [source][6], node[YOWAmP1SSD2IFm9IQ_0seA], [P], s[STARTED], a[id=z34owuW4TS2RQMJxeRUlJg], message [shard failure, reason [test]], failure [Exception[test]], markAsStale [true]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:312) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:03,629][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [RED] (reason: [shards failed [[source][6]] ...]). 1> [2018-05-05T20:41:03,751][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[source][6]] ...]). 1> [2018-05-05T20:41:04,030][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [target] creating index, cause [split_index], templates [], shards [128]/[0], mappings [] 1> [2018-05-05T20:41:04,185][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [target/Ru5mKdhxQpaT3gWmSTvH2w] create_mapping [type] 1> [2018-05-05T20:41:04,634][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][18]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:04,756][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][16]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:04,757][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][19]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:04,777][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][17]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:04,803][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][18], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=csv1Up2NRXuGv6ym8KS52g], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[no_attempt]], message [failed recovery], failure [RecoveryFailedException[[target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:04,804][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][16], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ZrUc_ooqTrKPM295EPD-_Q], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[no_attempt]], message [failed recovery], failure [RecoveryFailedException[[target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:04,805][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][19], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=jR2yVOyyTQW07l_ldZyi5w], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[no_attempt]], message [failed recovery], failure [RecoveryFailedException[[target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:04,806][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][17], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Bt7u2BuURz-eUmXmAGxAlg], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[no_attempt]], message [failed recovery], failure [RecoveryFailedException[[target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:04,820][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [RED] (reason: [shards failed [[target][18], [target][18], [target][16], [target][19], [target][17], [target][19], [target][16], [target][18], [target][17]] ...]). 1> [2018-05-05T20:41:05,256][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][22]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:05,266][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][20]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:05,275][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][23]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:05,292][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][21]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:05,867][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][22], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=3zIJDmNrSFG1a3mM5HRZBg], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:05,868][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][20], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ObvPh2O7RO-r29mogn6bnA], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:05,869][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][23], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=hGhXCRRhRmyD_iF4IfdHEw], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:05,870][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][21], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=gN9elTk6QqKhEhahN7lqHw], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:06,128][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][19]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:06,131][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][18]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:06,244][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][17]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:06,326][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][16]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:06,955][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][19], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=HVgcidBgTp26Q5QPVNelqw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:04.805Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:06,956][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][18], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ufRZS4ERTve8g4pF2IjWrg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:04.803Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:06,957][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][17], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=uC1Ao9V4SfWlgfL64-N_3g], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:04.805Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:06,958][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][16], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=I6yRwha1Tf-MJp51vKmH8w], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:04.804Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:07,338][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][22]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:07,404][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][23]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:07,732][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][21]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:07,743][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][20]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:08,184][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][22], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=VyN0TBW-T8W-eWne3FZDmQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:05.867Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:08,185][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][23], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=XTRLhg9cTTmvu9jhYgUkKQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:05.869Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:08,186][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][21], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Zyvo_JwcSl6BeGjiObr9EQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:05.870Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:08,187][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][20], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=BzJkms41RL-imO3Y5NOsuA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:05.868Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:08,895][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][17]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:09,142][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][19]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:09,166][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][16]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:09,188][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][18]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:09,391][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][17], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=hk6dUBgmTMSwx4zX7H80Mg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:06.957Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:09,392][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][19], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=STjERQ0pQyGZ06xm8wwGIQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:06.955Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:09,393][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][16], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=99I9pG4ySKW3C7-DH4QiPw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:06.958Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:09,394][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][18], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=B9o_0fAvTWqWAfNlRGH3Rg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:06.956Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:09,701][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][21]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:09,710][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][22]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:09,736][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][23]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:09,806][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][20]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,009][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][21], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=QyhfSYn-SKiMQPRkoGtwLQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:08.186Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,010][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][22], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=wQoJZ7FlSG6t1MsxfqMy2w], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:08.184Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,011][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][23], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=MltN0JyFRfOrr4tkD3NTgw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:08.185Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,012][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][20], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=MEx_G8qOR9eMShH8lU0MbA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:08.187Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,237][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][16]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,241][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][17]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,384][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][19]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,386][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][18]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,622][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][16], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=BgKIuUnbQLGs_HzPkNd0lA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:09.393Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,623][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][17], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=KVAyUfngRYabrLfZO5BUXQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:09.391Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,623][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][19], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ASJt4vdXSOO5N12rYntgJw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:09.392Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,624][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][18], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=9JtxSTCBTduN4FBBt3db3A], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:09.394Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,711][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][22]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,716][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][23]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,717][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][64]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,722][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][21]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,726][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][22], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=IQSdeQJMQym6ZTZJTkeaeA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.010Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,727][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][23], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=u1s8SlffQUa72v-1zZSntw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.011Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,731][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][64], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=vdEK8rvcSO2HRd1tHtNARg], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,733][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][20]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,774][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][65]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,787][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][67]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,800][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][66]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,803][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][21], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=y_bTlgyNQsee3aS4m0G0BA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.009Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,804][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][20], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=jpGM8zzETOeTdWmZW3nV7Q], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.012Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,804][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][65], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=CmVEQbPOST2NqKppCglaTg], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,805][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][67], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ppjr5teVS_yehPC_NRA-4A], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,806][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][66], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=2a3wilbTSYiD5kTLTvocBg], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,922][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][19]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,925][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][18]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,928][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][64]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,945][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][64], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=VQwwI-a8TEugoBzJvhjTLQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.731Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,946][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][19], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=WSRG4yxPT3KgQvA3J2XdmQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.623Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][19]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\19\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,947][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][18], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=d95f78WnQ0Gv2dqM3etP8Q], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.624Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][18]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\18\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,980][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][17]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,981][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][16]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:10,985][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][70]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,026][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][68]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,028][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][69]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,031][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][20]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,039][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][21]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,020][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][17], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=2_Wpu_XGRTCUohc7t9dlhA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.623Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][17]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\17\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,041][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][16], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=lXxRQxhnS52BBJ0rikZSFw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.622Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][16]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\16\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,041][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][70], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=R7nKjWMtQeWrBop81KY0Vg], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,052][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][65]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,090][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][68], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=YBDgu592RLSz0G5sHy-Mgg], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,091][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][69], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=nqDk5fFWTJibpVMEdmteCg], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,092][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][20], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=UqZAhoakSw6oRyjp5GWaFg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.804Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][20]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\20\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,093][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][21], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=gZjybrN_R7yM09fqYcIbrQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.803Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][21]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\21\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,094][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][65], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=B5-WXNhhQC6ZFYSgSadIoQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.804Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,301][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][70]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,343][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][64]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,357][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][24]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,359][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][25]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,366][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][66]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,375][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][22]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,384][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][23]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,393][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][67]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,400][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][70], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=KqLDmBUjTMSTV7v1NYvoHA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:11.041Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,401][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][64], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=QpWehGqHTa6jl60NQ1rQbA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.945Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,402][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][24], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=FfskeS5ZS-CivwUSPh-RRQ], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,403][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][25], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=39mI0BFEQaWj9DSmamzdJg], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,404][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][66], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ybntqa1hScqtzoqnbLBY8g], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.806Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,404][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][22], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=e_0nKy_KTOKlmYRLhy6eNQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.726Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][22]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\22\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,405][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][23], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=edNNJg9NTBaj1XQ_7GiUqg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.727Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][23]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\23\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,406][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][67], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=KG368nIoSE6n51CuTD1ctg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:10.805Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,632][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][69]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,635][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][65]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,656][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][27]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,687][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][26]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,717][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][69], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Vd4C5RLMR1-6RXtVk3xMBw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:11.091Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,718][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][65], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=vnSqm-6OQVKv5JQemQKVSQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:11.094Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,719][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][27], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=95JofwiHQZS8RNT3TOXgyw], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:11,722][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][26], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=H5l0apjPQQKyQ5ZSb1Ytqg], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,102][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][68]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,146][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][28]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,148][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][29]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,176][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][71]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,313][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][68], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=gkEvIrKVQCe4A9EQnC11Pg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:11.090Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,314][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][28], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=F6a58GLJRIS99QvOrdMRQQ], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,314][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][29], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=xVs_0rylSL26Usjxd4sRSA], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,315][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][71], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=z2YnrIIoQ0qcvA-BxuP6qQ], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,649][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][25]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,653][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][26]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,657][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][64]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,659][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][24]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,660][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][67]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,661][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][65]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,696][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][25], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Un64haN1SgqkQTaG9lKn7w], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:11.403Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,697][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][64], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=4rLZmdjjTmemsFzGbtWPYQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:11.401Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,698][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][26], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=99IQ7qZeSTO5wX33PvCp4A], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:11.722Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,699][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][24], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=9FaFMJRVRE24igEu4CNjsA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:11.402Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,700][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][67], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Xy2dSJ5CRwaYxbmiZaJvGw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:11.406Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,700][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][27]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,701][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][65], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=YdBFkPuETi6Gpecu5nXfiA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:11.718Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:12,743][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][66]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,238][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][27], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=5OtgrJxuRAe-p_ZG8kEAQA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:11.719Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,239][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][66], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Zzkifbj1SwWmAsWrIt3nYw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:11.403Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,375][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][70]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,415][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][70], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=kR9nYCYURXm-WqPJWv6-8A], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:11.400Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,459][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][64]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,464][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][64], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=rm9X8EV9TwuinY9hlJ7EKw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:12.697Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][64]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\64\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,476][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][65]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,493][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][65], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=cgaAQx4cRDGEsA7e35eM8A], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:12.700Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][65]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\65\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,514][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][29]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,521][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][24]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,534][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][29], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=8N58OoXOSFa5j3SGGTW0Wg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:12.314Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,619][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][67]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,623][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][24], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=g9oH2AGpTCKqVcBJsdE3sg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:12.699Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,668][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][28]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,669][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][67], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=mhG0eiAYRO6t_3Vs06Thug], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:12.700Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,681][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][66]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,754][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][30]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,760][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][66], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ocQEyaWGTUO2Ql1y6JWOmg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:13.239Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,761][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][28], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=csFmCKvqTbmHD4swA_DsFg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:12.314Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,762][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][30], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ZAZANEchTzyjvy0ItXzF1A], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,803][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][25]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,820][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][25], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Yyq7QR3VRsK7ahhhLbWKnA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:12.696Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,822][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][68]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,826][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][26]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,829][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][69]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,832][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][24]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,839][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][27]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,861][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][70]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,877][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][68], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=C0wdMD5xQPyynwinSbX3Xg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:12.312Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,878][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][69], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=G6fFpfctQX620Zw1OaMPUw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:11.717Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,879][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][26], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=HA-x4HfATby7t0kY4HXMwA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:12.698Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,880][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][24], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=4yqOYwyqTySMo_LIQWS53A], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:13.623Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,880][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][27], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=GX5dBXT3S42U8g91tBzsVw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:13.238Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:13,881][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][70], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=LIRnocbARFW1GhCWL0GDMA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:13.415Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,001][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][28]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,004][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][25]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,010][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][66]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,032][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][72]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,041][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][67]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,052][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][29]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,060][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][30]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,068][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][71]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,073][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][66], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=8exKWh-dSyODzzcehBT1Eg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:13.760Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][66]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\66\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,074][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][72], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=_Jx1Cve6Q5y-hpCs6_OkGQ], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,075][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][28], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=8RNILvGIQim7uVHMjWBdGQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:13.761Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,076][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][25], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=4EJ_P_CeSQyO20OkGl5M_Q], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:13.820Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,076][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][67], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=yiCx-cDoTnO36q45NyP45w], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:13.669Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][67]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\67\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,077][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][29], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=nJ9VhjTYQny6pfLPWr_VNA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:13.534Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,078][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][30], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=6DUpS5pZSLS-sgn8AtidzQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:13.762Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,079][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][71], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=7rGSXuwXQTWfDweClrzNmg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:12.315Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,234][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][24]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,258][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][73]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,282][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][69]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,308][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][24], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=CpsRrvFwTuSUpPfDBYFNNA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:13.880Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][24]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\24\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,309][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][73], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=mKtEP4tHQdulC8Oqzzwm7g], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,310][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][69], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=4JelPiQ9Q3OkL5JZAx9-9A], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:13.878Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,347][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][70]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,355][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][68]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,368][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][26]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,372][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][27]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,400][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][70], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=KjcKV9UfSvG1AP3MrWyOYA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:13.881Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][70]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\70\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,400][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][68], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=TZxAfs46TVGohtYbDYZZ0A], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:13.877Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,401][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][26], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=E4ZOLDoWTcycahvAK6sYdQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:13.879Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,402][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][27], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=SRpAB9WvSGCzZFUDMKXbmQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:13.880Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,543][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][71]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,546][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][28]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,550][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][71], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=IV1Pb1efSK2PHCqXIqThOA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.079Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,551][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][28], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Fia4jnn-THCzAZzDa7EUOA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.075Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,557][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][25]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,566][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][72]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,568][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][29]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,573][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][73]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,576][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][69]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,578][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][26]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,596][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][25], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=8TjZvAQdTd-JLTf8GurhSw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.076Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][25]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\25\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,597][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][72], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=LtqjAQaERBePyC2rCitLHg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.074Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,598][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][29], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Ef5OqKmjSKS57N_nIgCrrw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.077Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,599][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][73], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=A-arI9_mSw242pJvb1wSBQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.309Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,600][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][69], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=mXn2mPGbSHqcQVswSgXLLQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.310Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][69]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\69\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,601][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][26], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=1oYWXLWlTCK8jQBkomZbLQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.401Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][26]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\26\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,747][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][75]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,748][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][30]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,751][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][71]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,765][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][28]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,768][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][27]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,768][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][68]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,803][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][75], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=p9U29K46TheV0nuscaZBQQ], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,804][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][30], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=1DGJoQGES2eVkO7T1_Ld7g], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.078Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,805][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][71], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=jjEbCGifQBOAsysT6pjaSw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.550Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,806][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][28], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ymdketwNRvquLYMsn0KR9Q], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.551Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][28]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\28\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,807][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][68], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=djat8vWwT3WBaiFMrq5pBg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.400Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][68]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\68\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,808][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][27], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=zLymOplFTIe_tSqIEZ248g], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.402Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][27]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\27\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,982][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][72]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:14,983][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][72], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=JQjMIMJzS0WpBDRRPBf30g], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.597Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:15,007][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][71]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:15,012][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][29]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:15,037][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][73]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:15,113][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][71], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Po0Daao3Sg-KKWWajdzZKg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.805Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][71]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\71\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:15,114][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][29], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=rwVWxG__SX-Ts6H2x5_p4w], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.598Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][29]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\29\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:15,115][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][73], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=f_CGiBbVQ1qYdEwXclcHXw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.599Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:15,222][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][76]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:15,540][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][72]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:15,621][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][76], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=8D6fNj_rTRunnNGejC0LMw], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:15,622][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][72], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=nxf8Ien_RvqJbHzCrfa3EQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.983Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:15,672][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][30]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:15,673][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][75]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:15,842][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][77]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:15,865][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][30], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=S3obOshbSqCYFIsOWsj5lA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.804Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:15,865][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][75], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=V8tPmu8-StOvzqslwkWm7A], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:14.803Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:15,866][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][77], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=XJgktpCfQka9WrPvoBhGcA], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:16,180][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][30]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:16,192][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][73]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:16,194][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][73], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=56AEgZFMQMGrRMlLmVrzlQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:15.115Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:16,246][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][72]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:16,251][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][76]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:16,295][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][78]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:16,304][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][30], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=EMycLNUKQM6JDNfuKCqVrA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:15.864Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][30]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\30\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:16,307][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][72], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=I3PBVPNGT5C43RgYM-qL-w], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:15.622Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][72]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\72\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:16,308][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][76], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=DIuO1r1fRpK1oyID9PZnfw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:15.620Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:16,309][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][78], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=e3WP9lf-TfagkPJphjoQqQ], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:16,548][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][77]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:16,556][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][73]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:16,600][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][77], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=lyoNEwUESN6_RNAPrczVNQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:15.866Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:16,601][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][73], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=3z-D_wqeQbKhq2sYUlJQBA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:16.193Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][73]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\73\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:16,684][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][79]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,013][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][75]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,200][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][79], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=mTrghVhCSruB8IfuFYFlXQ], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,201][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][75], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=tWxOZuviQDqnf1sJ-4vfUA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:15.865Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,340][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][77]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,355][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][78]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,358][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][76]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,398][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][81]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,446][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][80]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,460][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][77], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=eGiJdV5gREG6a3ZU1oKPhg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:16.600Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,461][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][78], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=0lxI0llaRs270jd5kv7xEQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:16.309Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,462][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][76], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=2qhhB5QxQQKVmFUGDZ9URQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:16.308Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,463][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][81], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=tXgThToCQ26z-laqHmRkfw], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,463][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][80], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=LIt-m4-lTpmz5E2bI6Lh9w], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,593][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][96]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,667][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][96], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=l5I2eGm6T4iAwUElCObVjA], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,697][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][79]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,700][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][81]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,703][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][75]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,730][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][97]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,761][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][82]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,812][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][80]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,842][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][83]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,852][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][79], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=9um4U9ObSWCY4Io1oEV8Kg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.200Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,853][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][81], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Ix7xErKcQ8mRHiWowyDZng], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.463Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,853][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][75], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=WyPEkGCaQE6SHJZZ0JdmdQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.200Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,854][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][97], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=wzwceGCeRmGsbQPltegi2A], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,854][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][82], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ZIeZuAuISUuSRByPtwtDRA], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,855][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][80], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=kCBX-89-SDOaGCV2TD8XtQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.463Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,856][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][83], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=qzWtg0C9Rs6e1Gips1886A], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,948][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][78]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,969][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][96]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,976][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][76]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,979][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][78], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=e9ZOUOLiSZq6LTOfic2B2Q], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.461Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,980][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][96], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=mACobhFxQTeZTkDZbyM16A], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.667Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,981][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][77]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:17,981][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][76], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=LCD0wn5BRHCTLicD3TWbHA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.462Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,022][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][85]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,073][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][86]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,074][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][87]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,075][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][77], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=boeQFE00SbKuIk2H3CRA7w], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.460Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,076][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][85], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Fzg2l8wBSbih45MwxJF93Q], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,077][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][86], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=yU_Vau8oSs-b1GLCqbWbNw], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,160][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][87], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=5MkzErzDRUGR8qgLfc2zQA], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,173][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][97]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,188][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][75]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,189][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][81]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,191][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][80]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,197][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][79]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,199][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][82]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,205][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][76]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,220][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][97], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=tKOzn84bQVeBAm-E1WLRZA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.854Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,221][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][75], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Pku_fRH-Q12Avnv1YUjbtg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.853Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][75]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\75\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,222][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][81], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=INqWMLl5TjK2M5gljhES-w], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.853Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,223][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][80], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=MhWof_mPTA-1x0ZYIrEcTA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.855Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,224][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][79], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=oATjLeBBQFe1IQEelqEjfA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.852Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,225][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][82], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=GM2OxVpwTJ-iyG4rMJeS2w], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.854Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,225][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][76], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=_O-GOdZwQeyeTxrD8OSCsA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.981Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][76]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\76\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,326][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][96]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,332][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][96], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=wWTxwAERR9e3ljxpHXgLZw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.980Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,333][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][87]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,335][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][86]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,340][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][85]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,345][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][98]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,367][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][78]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,374][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][77]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,396][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][83]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,399][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][87], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=TrDNTB1yShS-gIvth6TJsg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.160Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,400][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][86], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=YxSLTq8IStqPnWjWZsjmzA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.077Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,401][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][85], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=sgnH_eXkRi-2VjU2c1QzhA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.076Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,402][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][98], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=xgwp8XF5QdemN1kICoqGgw], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,402][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][78], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=iYydwh1uT6m_Dd6sLESGIw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.978Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,403][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][77], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ksRRKTd3RUWHE4HGpcJVXw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.075Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][77]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\77\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,404][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][83], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=WswJJmBATxqM011Yl81MLw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:17.856Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,506][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][97]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,514][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][96]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,522][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][82]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,530][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][80]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,543][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][79]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,546][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][97], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=UPJyoz1ORDCYoTlZJTJnhw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.220Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,546][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][96], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=8FVzwepZSWy6CC1qRDZqnw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.332Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,547][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][82], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=BvSuUuk0Sqm8Au7wFyzUZg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.225Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,548][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][80], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=pJozcYAxT2ejibEXyfucHw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.223Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,549][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][79], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=2IuMg5LAQ_ejesPlSF0wyg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.224Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,579][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][88]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,581][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][81]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,614][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][99]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,618][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][88], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=AC6EZkZyT-amC_7HswsZ1w], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,619][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][81], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=UXlPn6FsTVyOQkrv0LU5TQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.222Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,620][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][99], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=CuAhCrCIQ5W7MopXgOPxqQ], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,630][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][85]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,633][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][78]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,639][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][98]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,653][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][83]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,677][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][100]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,683][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][85], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=FXqXoAq9Rg2xij2pez5ZbQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.401Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,684][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][78], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=M5fQ7O2fTdmeyHaULtb4QA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.402Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][78]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\78\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,685][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][98], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Nu2Yad3ASJaf62mLeBs9Tw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.402Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,686][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][83], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=QWnp-F-pQUuOHT7l0q55Cg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.404Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,686][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][100], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=sbTxyx_LSUaC6JSx-HGOIg], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,745][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][80]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,751][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][96]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,753][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][82]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,753][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][80], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=cUjLxDRrSje2cU7Tqb69tA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.548Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][80]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\80\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,754][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][99]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,754][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][96], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=hF_cmBG6S9WTiP4HSJWa3w], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.546Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][96]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\96\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,755][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][79]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,756][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][81]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,757][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][86]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,758][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][97]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,782][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][82], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Y3vDHJKZS3-Fu6nut79X0w], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.547Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,783][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][99], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=6anm2KXsQsyOfZGy7A_fEQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.620Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,784][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][79], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=C6nha4n-TnKwLxrKEC-g7g], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.549Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][79]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\79\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,784][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][81], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=0Eh9ghAnScemKY3jdvbdaA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.619Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][81]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\81\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,785][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][86], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=RAeoujxOTeKtMzbVE0pXVg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.400Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,786][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][97], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=bjKxo_PgQeGOSb3oy8yxsA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.546Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,857][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][98]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,860][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][85]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,863][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][83]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,864][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][88]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,865][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][100]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,873][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][98], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=JWiQtzODTjuV_DPSQYMutw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.685Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,874][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][85], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=JQV1ce6ZRLu5tSU24u5Irg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.683Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,875][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][83], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=gopoFa0OSOyk3bUX6XvEhA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.686Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,876][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][88], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=nLR2jkMeTIKl7FCprfzIVQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.618Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,876][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][100], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=DpPTcBwbRUGYP2h5aQXpoA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.686Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,875][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][87]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,902][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][101]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,959][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][87], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=bTJjy1JpTe-7OXCePNLXIw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.399Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:18,960][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][101], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ANL9PM20SBavVDc8x-QgbQ], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,017][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][99]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,018][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][99], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Vv9wNd7_TIexmQe3axBIfg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.783Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,029][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][97]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,031][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][100]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,038][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][98]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,042][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][82]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,046][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][83]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,050][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][86]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,055][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][89]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,061][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][97], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=A6DR8zQJQcy9ODjMBnzgiw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.786Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][97]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\97\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,062][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][100], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=K9E4Pzd6SFKy_vPddtYbrA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.876Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,062][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][98], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=WFaXSzMvSoihqJF7HGpxEg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.873Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,063][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][82], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=xixnJsKDRkq8uih8K3F8wQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.782Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][82]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\82\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,063][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][83], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=a03tsNWjRYqjHdsQVdKoNA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.875Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][83]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\83\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,064][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][86], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=HiUX5XfpRlCldxfPkZfeYQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.785Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,064][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][89], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=NHaRiUxnRciocdVP47n2OQ], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,144][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][99]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,149][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][87]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,154][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][88]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,163][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][99], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=X9ku7-3cQTePZfiPPNYiwQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.018Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,163][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][85]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,164][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][87], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=_8WJK2K4TYK5oELrE4zoYg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.959Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,165][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][88], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ugD5NhMKQk2iY_7ncG9n9w], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.876Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,166][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][101]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,182][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][104]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,196][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][90]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,200][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][103]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,204][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][85], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=axmKmdoaQeOBjV-L72qhXA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.874Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][85]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\85\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,204][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][101], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=tNHOar0-RbCOPVqV7x4i1g], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:18.960Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,205][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][104], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=vDGJQhHCQ8OsKLdt5luH7A], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,206][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][90], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=P6-XjsJGSHaL5biQ1LHqdg], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,206][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][103], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=JX7PzQcSSh2DdLWPOWGV7Q], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,278][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][98]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,281][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][99]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,284][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][86]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,293][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][89]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,296][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][88]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,300][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][87]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,301][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][98], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=J4WI4TvhT3ql1WKop96Fhw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.062Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][98]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\98\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,302][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][99], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=eDsLp2-yT7KVOvnYHKOSdA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.163Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][99]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\99\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,303][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][86], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=AHooOATwRvG_U7unNS8wPA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.064Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][86]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\86\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,304][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][89], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=fKxhme5sRpGkjwM2k4QmUQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.064Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,312][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][88], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=2iB2w_ESRcmSMJgt77sKxg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.165Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,337][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][105]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,351][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][100]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,406][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][87], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=C6SlgkgGQDGwR5xtXsckJA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.164Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][87]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\87\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,407][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][105], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=VvjXjTS7R_utZzEEgNNb7g], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,407][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][100], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=leB7P2RjSfSaocp53oPbqg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.062Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,517][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][92]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,525][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][101]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,533][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][88]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,541][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][90]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,566][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][103]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,581][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][91]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,609][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][106]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,612][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][104]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,619][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][92], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=3VHj2PUhTEeJKjmBDzRqEQ], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,619][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][101], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=kpYyE9vGReWL6Y-9pe7D2A], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.204Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,620][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][88], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=HzvZCnw3S7-QkgOxZWEWXQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.312Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][88]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\88\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,621][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][90], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=wIrc7OWuTgKy_lcWUkFrRQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.205Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,622][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][103], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=DSBMJ-tpSbKpBJSpcnfbIw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.206Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,623][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][91], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=mMQwaGgBQH6Ou-KED_4AuA], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,624][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][106], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=UX_mYBPMQ5edFDMK10bM7w], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,625][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][104], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=8qKvrH2TTk2vvNSpEN0V_A], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.205Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,743][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][100]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,746][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][105]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,749][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][89]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,766][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][100], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=IBfqVQGqQ76SNavT9SgeGA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.407Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][100]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\100\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,768][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][105], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=6sjuOM6JQH6BJ9Y3SM1Pug], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.407Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,769][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][89], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=E_bcAQuUSEObw4e6bD8sHg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.304Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,814][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][107]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,822][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][93]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,823][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][95]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,863][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][107], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=8sDksA0LRRCpc3S2uLF38Q], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,864][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][93], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=9LKTYO2LSCeEvqzXtzZMnQ], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,865][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][95], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=E5UxSsYRTrKp-JC35k4ndw], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,871][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][94]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,883][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][108]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,924][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][91]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,934][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][90]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,935][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][89]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,935][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][104]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,938][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][101]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,948][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][103]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,952][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][94], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=SyayTgUQRYKE1wyd93LQlg], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,953][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][108], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=xU1_Cr75Q2SeNAOFTQNQPA], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,956][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][91], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ScEZjzLKSQ-Cx9CRdXd7bg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.623Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,958][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][90], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=6X-_u3RVSayVy6GJE6nE3Q], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.621Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,959][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][89], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=mxX_r836TZ60lYgUraU-JA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.769Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,960][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][104], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=d_FPlRpnSBKoSoKknT1FKQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.624Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,960][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][101], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=icZjHDzVQ7qbT3RTFaLsMQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.619Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:19,962][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][103], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=UlhTi32kQxKfTSGiiTDIzQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.622Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,089][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][95]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,091][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][93]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,095][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][92]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,098][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][107]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,102][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][106]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,116][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][95], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=98pV1RsUROGWvHiSBR5j3w], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.865Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,119][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][93], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=tpi1APcdSh2DbEtGM7w2iw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.864Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,119][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][92], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=6woGdHtgRpyIobSNjAwvpw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.619Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,120][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][107], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=6NJnUfaOQdmKebEPH82fQg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.863Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,122][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][106], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=rEOmNQyoTJuJLWUVGsaCOQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.624Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,130][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][105]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,138][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][109]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,217][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][105], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=XkMkYbbIQ2aAveBb7l9Flg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.768Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,218][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][109], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=zW8iCqBmSa2aGRaspb5ziA], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,276][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][92]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,276][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][104]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,278][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][91]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,278][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][106]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,279][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][90]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,285][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][103]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,285][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][89]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,286][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][92], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=162OGm59T9W2dahZVQCeIg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.119Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,287][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][104], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=QxSMc0E4Qgula8dvmPErpw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.960Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,288][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][91], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ocvhcQiSSIi7cK5wpfEnbg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.956Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,289][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][101]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,289][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][106], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=gAopaIiKQ-mxNy5fxnX8aQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.121Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,290][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][90], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=IU2oSmcnTHeYIe-qA-BShg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.958Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,331][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][103], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=oo-KgU2MTmGz-SumYnHFcA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.961Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,331][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][89], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=K2oBblsfR8yhrAQVsd5x8w], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.959Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][89]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\89\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,332][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][101], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=yoi6SjRrRsmokIyHMOELJA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.960Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][101]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\101\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,388][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][90]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,391][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][104]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,405][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][93]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,412][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][107]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,413][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][95]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,415][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][105]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,416][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][90], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=RxxEu81vRs-nF_aDndUzZw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.290Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][90]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\90\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,417][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][104], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=9YnEvM3DSCiFnTLkHA3UNA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.287Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][104]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\104\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,418][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][93], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=vvpZeRsHRoy1IhJ7CWkrLw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.119Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,417][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][94]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,421][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][107], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=qaxt35JATuaiHVrFsxrYSA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.120Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,422][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][95], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ETkUveLJSAGsT_KAbpr0RQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.116Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,423][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][105], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=90idvPDwTg6Kwh6RNdJXPw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.217Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,424][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][106]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,474][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][94], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=FKrglzWuQgajeobjlVWX4A], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.952Z], failed_attempts[1], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,474][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][106], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=bKg1ibXGRjGbKRA7M7AYgw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.289Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,519][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][91]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,523][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][105]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,525][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][109]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,526][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][91], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=pTmdbdV8TNy7Fn4oRBqO5A], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.288Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,527][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][105], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ZIFe_sMyQuapJqkRGy9E1g], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.423Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][105]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\105\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,528][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][109], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=E9c4FvSlQAKXLzYXy7I8dw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.218Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,528][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][103]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,529][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][93]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,532][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][92]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,535][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][108]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,585][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][103], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=_l8zX9XhTaKCXJxCXwXEjg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.331Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][103]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\103\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,586][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][93], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=rbNKz8V2RQGMsQjM-OvKVw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.418Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,587][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][92], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=AT0kTiMZTKSdGbco1noGEw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.286Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,588][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][108], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=uBiz6AkST72E8m6EtHQSWQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:19.953Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,642][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][107]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,656][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][109]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,657][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][106]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,665][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][107], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=w7vU8ulwRjSsSmeQ-xdw2A], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.419Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,666][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][109], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=a-HwzCxkQRqhqWO5sy_j9w], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.528Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,667][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][106], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=Q70xmpsxTFCGBkVU0ekxIw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.474Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][106]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\106\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,684][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][91]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,693][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][95]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,731][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][110]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,745][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][94]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,746][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][91], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=F2Zpt-3JSwaUOJQhbKO12A], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.526Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][91]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\91\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,747][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][95], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=AgXZvFFjS3C1FaKHT2xHrg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.422Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,748][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][110], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=93lpAa16QtqxR3iJak8WPQ], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,828][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][94], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=VMpvHwkFTIaxQfFD8kL-uQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.474Z], failed_attempts[2], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,841][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][93]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,881][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][109]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,886][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][107]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,887][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][111]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,890][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][108]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,892][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][92]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,893][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][95]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,900][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][93], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=KgKqxilDQnGLGLNoeSRQuw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.586Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][93]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\93\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,901][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][109], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=OviBNLQMRqW0NYzXGjKmhw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.666Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,902][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][107], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=BkTsIqrpTe-ORE6q9jExNQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.665Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][107]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\107\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,903][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][111], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=p9kMbbmsSuCG_IoV9-yUBw], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:41:04.030Z], delayed=false, allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store.newIndexWriter(Store.java:1635) ~[main/:?] 1> at org.elasticsearch.index.store.Store.bootstrapNewHistory(Store.java:1487) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:393) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:133) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,903][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][108], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=P_xoydrrTumjScSy9bZ5gw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.588Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,904][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][92], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=2dJEUH1tSVCcZs2wq-BRyQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.587Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][92]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\92\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,905][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][95], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=I8HuTguNRWyfvTK63C2b5Q], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.747Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][95]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\95\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,962][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][94]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:20,967][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][110]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,003][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][94], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=bmbJpigJT8iAMITgB2VLFA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.827Z], failed_attempts[3], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,004][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][110], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=ILWD6j0uS624mcEV-8vgBA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.748Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,125][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][109]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,128][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][110]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,129][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][94]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,130][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][108]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,131][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][109], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=LQGx-_bFQyi6Pjym_XX-EA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.901Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][109]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\109\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,131][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][110], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=A8hAm1B0ShW6sbEMplw0NA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:21.004Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,132][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][94], node[mIq8KinYToaJSuyTqY1_hQ], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=YALj95HTRvCOD7ozdUwkyQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:21.003Z], failed_attempts[4], delayed=false, details[failed shard on node [mIq8KinYToaJSuyTqY1_hQ]: failed recovery, failure RecoveryFailedException[[target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][94]: Recovery failed on {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\Ru5mKdhxQpaT3gWmSTvH2w\94\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,133][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][108], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=4xshOLK8TqqZwIaAGx0M9Q], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.903Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,139][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][111]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,185][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][111], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=HskY7fURS16e5a9VytB5Vw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:20.903Z], failed_attempts[1], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,318][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][110]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,322][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][110], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=O3-SzT3wQRSDOAWIO_S92g], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:21.131Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,323][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][108]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,337][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][111]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,362][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][108], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=uG665jU-R4297VX2a7QZvA], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:21.133Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][108]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\108\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,363][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][111], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=oCXo4BLASrOCxDyec76Srw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:21.185Z], failed_attempts[2], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,494][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][110]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,495][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][110], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=h8z94c_XQmaqlw9hZDa05g], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:21.322Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][110]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\110\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,513][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][111]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,522][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][111], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=SHEyFripQmmNaFtX96wLqg], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:21.363Z], failed_attempts[3], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,661][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[target][111]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:21,662][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][111], node[YOWAmP1SSD2IFm9IQ_0seA], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=YdcSS5JHSpaHvJOq8dSDbw], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:41:21.521Z], failed_attempts[4], delayed=false, details[failed shard on node [YOWAmP1SSD2IFm9IQ_0seA]: failed recovery, failure RecoveryFailedException[[target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], allocation_status[deciders_no]], message [failed recovery], failure [RecoveryFailedException[[target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [target][111]: Recovery failed on {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2047) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:343) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> Caused by: java.lang.IllegalArgumentException: Directory StatsDirectoryWrapper(HardlinkCopyDirectoryWrapper(store(MMapDirectory@C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Ru5mKdhxQpaT3gWmSTvH2w\111\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@66502081))) still has pending deleted files; cannot initialize IndexWriter 1> at org.apache.lucene.index.IndexWriter.(IndexWriter.java:698) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:169) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:131) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301) ~[main/:?] 1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:123) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1520) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2042) ~[main/:?] 1> ... 4 more 1> [2018-05-05T20:41:34,159][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testSplitIndexPrimaryTerm]: cleaning up after test 1> [2018-05-05T20:41:34,337][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [source/vHlv_SWpSoWaq6CvE9K2fw] deleting index 1> [2018-05-05T20:41:34,337][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [target/Ru5mKdhxQpaT3gWmSTvH2w] deleting index 1> [2018-05-05T20:41:34,464][WARN ][o.e.i.IndicesService ] [node_s1] [source/vHlv_SWpSoWaq6CvE9K2fw] failed to delete index 1> java.io.IOException: could not remove the following files (in the order of attempts): 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\vHlv_SWpSoWaq6CvE9K2fw\1\index\_0.cfs: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\vHlv_SWpSoWaq6CvE9K2fw\1\index\_0.cfs 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\vHlv_SWpSoWaq6CvE9K2fw\1\index: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\vHlv_SWpSoWaq6CvE9K2fw\1\index 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\vHlv_SWpSoWaq6CvE9K2fw\1: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\vHlv_SWpSoWaq6CvE9K2fw\1 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\vHlv_SWpSoWaq6CvE9K2fw\5\index\_0.cfs: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\vHlv_SWpSoWaq6CvE9K2fw\5\index\_0.cfs 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\vHlv_SWpSoWaq6CvE9K2fw\5\index: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\vHlv_SWpSoWaq6CvE9K2fw\5\index 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\vHlv_SWpSoWaq6CvE9K2fw\5: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\vHlv_SWpSoWaq6CvE9K2fw\5 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\vHlv_SWpSoWaq6CvE9K2fw: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\vHlv_SWpSoWaq6CvE9K2fw 1> at org.elasticsearch.core.internal.io.IOUtils.rm(IOUtils.java:187) ~[elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectoryUnderLock(NodeEnvironment.java:505) ~[main/:?] 1> at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectorySafe(NodeEnvironment.java:489) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.deleteIndexStoreIfDeletionAllowed(IndicesService.java:668) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.deleteIndexStore(IndicesService.java:655) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:561) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:284) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:218) ~[main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:496) ~[main/:?] 1> at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_162] 1> at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:493) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:480) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:431) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:161) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:35,270][WARN ][o.e.i.IndicesService ] [node_s0] [source/vHlv_SWpSoWaq6CvE9K2fw] failed to delete index 1> java.io.IOException: could not remove the following files (in the order of attempts): 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\vHlv_SWpSoWaq6CvE9K2fw\4\index\_1.cfs: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\vHlv_SWpSoWaq6CvE9K2fw\4\index\_1.cfs 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\vHlv_SWpSoWaq6CvE9K2fw\4\index: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\vHlv_SWpSoWaq6CvE9K2fw\4\index 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\vHlv_SWpSoWaq6CvE9K2fw\4: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\vHlv_SWpSoWaq6CvE9K2fw\4 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\vHlv_SWpSoWaq6CvE9K2fw\6\index\_1.cfs: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\vHlv_SWpSoWaq6CvE9K2fw\6\index\_1.cfs 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\vHlv_SWpSoWaq6CvE9K2fw\6\index: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\vHlv_SWpSoWaq6CvE9K2fw\6\index 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\vHlv_SWpSoWaq6CvE9K2fw\6: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\vHlv_SWpSoWaq6CvE9K2fw\6 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\vHlv_SWpSoWaq6CvE9K2fw: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\vHlv_SWpSoWaq6CvE9K2fw 1> at org.elasticsearch.core.internal.io.IOUtils.rm(IOUtils.java:187) ~[elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectoryUnderLock(NodeEnvironment.java:505) ~[main/:?] 1> at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectorySafe(NodeEnvironment.java:489) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.deleteIndexStoreIfDeletionAllowed(IndicesService.java:668) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.deleteIndexStore(IndicesService.java:655) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:561) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:284) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:218) ~[main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:496) ~[main/:?] 1> at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_162] 1> at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:493) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:480) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:431) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:161) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:35,291][WARN ][o.e.i.IndicesService ] [node_s0] [source/vHlv_SWpSoWaq6CvE9K2fw] still pending deletes present for shards [[[source/vHlv_SWpSoWaq6CvE9K2fw]], [[source/vHlv_SWpSoWaq6CvE9K2fw]][4], [[source/vHlv_SWpSoWaq6CvE9K2fw]][6]] - retrying 1> [2018-05-05T20:41:35,313][WARN ][o.e.i.IndicesService ] [node_s0] [source/vHlv_SWpSoWaq6CvE9K2fw] still pending deletes present for shards [[[source/vHlv_SWpSoWaq6CvE9K2fw]], [[source/vHlv_SWpSoWaq6CvE9K2fw]][4], [[source/vHlv_SWpSoWaq6CvE9K2fw]][6]] - retrying 1> [2018-05-05T20:41:35,343][WARN ][o.e.i.IndicesService ] [node_s0] [source/vHlv_SWpSoWaq6CvE9K2fw] still pending deletes present for shards [[[source/vHlv_SWpSoWaq6CvE9K2fw]], [[source/vHlv_SWpSoWaq6CvE9K2fw]][4], [[source/vHlv_SWpSoWaq6CvE9K2fw]][6]] - retrying 1> [2018-05-05T20:41:35,394][WARN ][o.e.i.IndicesService ] [node_s0] [source/vHlv_SWpSoWaq6CvE9K2fw] still pending deletes present for shards [[[source/vHlv_SWpSoWaq6CvE9K2fw]], [[source/vHlv_SWpSoWaq6CvE9K2fw]][4], [[source/vHlv_SWpSoWaq6CvE9K2fw]][6]] - retrying 1> [2018-05-05T20:41:35,489][WARN ][o.e.i.IndicesService ] [node_s0] [source/vHlv_SWpSoWaq6CvE9K2fw] still pending deletes present for shards [[[source/vHlv_SWpSoWaq6CvE9K2fw]], [[source/vHlv_SWpSoWaq6CvE9K2fw]][4], [[source/vHlv_SWpSoWaq6CvE9K2fw]][6]] - retrying 1> [2018-05-05T20:41:35,689][WARN ][o.e.i.IndicesService ] [node_s0] [source/vHlv_SWpSoWaq6CvE9K2fw] still pending deletes present for shards [[[source/vHlv_SWpSoWaq6CvE9K2fw]], [[source/vHlv_SWpSoWaq6CvE9K2fw]][6]] - retrying 1> [2018-05-05T20:41:36,478][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [random_index_template] 1> [2018-05-05T20:41:36,493][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testSplitIndexPrimaryTerm]: cleaned up after test 1> [2018-05-05T20:41:36,493][INFO ][o.e.a.a.i.c.SplitIndexIT ] [testSplitIndexPrimaryTerm]: after test 2> REPRODUCE WITH: gradlew :server:integTest -Dtests.seed=734B52BB31887C24 -Dtests.class=org.elasticsearch.action.admin.indices.create.SplitIndexIT -Dtests.method="testSplitIndexPrimaryTerm" -Dtests.security.manager=true -Dtests.locale=sq -Dtests.timezone=Antarctica/McMurdo FAILURE 34.6s J1 | SplitIndexIT.testSplitIndexPrimaryTerm <<< FAILURES! > Throwable #1: java.lang.AssertionError: ResizeResponse failed - index creation acked but not all shards were started > at __randomizedtesting.SeedInfo.seed([734B52BB31887C24:81D1391557FCD0C0]:0) > at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked(ElasticsearchAssertions.java:127) > at org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm(SplitIndexIT.java:344) > at java.lang.Thread.run(Thread.java:748) 1> [2018-05-05T20:41:36,548][INFO ][o.e.a.a.i.c.SplitIndexIT ] [testCreateSplitIndex]: before test 1> [2018-05-05T20:41:36,549][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testCreateSplitIndex]: setting up test 1> [2018-05-05T20:41:36,571][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [random_index_template] for index patterns [*] 1> [2018-05-05T20:41:36,586][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testCreateSplitIndex]: all set up test 1> [2018-05-05T20:41:36,586][INFO ][o.e.a.a.i.c.SplitIndexIT ] using custom data_path for index: [KizLuaENTu] 1> [2018-05-05T20:41:36,586][INFO ][o.e.a.a.i.c.SplitIndexIT ] using custom data_path for index: [MmHPJSThcp] 1> [2018-05-05T20:41:36,591][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [source] creating index, cause [api], templates [random_index_template], shards [1]/[1], mappings [] 1> [2018-05-05T20:41:36,770][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [source/ptLmt_0ZT-il5H1Rwd6k2A] create_mapping [type] 1> [2018-05-05T20:41:37,020][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][0]] ...]). 1> [2018-05-05T20:41:37,087][INFO ][o.e.c.s.ClusterSettings ] [node_s1] updating [cluster.routing.rebalance.enable] from [all] to [none] 1> [2018-05-05T20:41:37,091][INFO ][o.e.c.s.ClusterSettings ] [node_s0] updating [cluster.routing.rebalance.enable] from [all] to [none] 1> [2018-05-05T20:41:37,098][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [target] creating index, cause [split_index], templates [], shards [2]/[1], mappings [] 1> [2018-05-05T20:41:37,180][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [target/bxh3kPxPS8KpzQt53gOX9Q] create_mapping [type] 1> [2018-05-05T20:41:37,544][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[target][1]] ...]). 1> [2018-05-05T20:41:37,566][INFO ][o.e.a.a.i.c.SplitIndexIT ] split node {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} 1> [2018-05-05T20:41:37,923][INFO ][o.e.c.s.ClusterSettings ] [node_s1] updating [cluster.routing.rebalance.enable] from [none] to [all] 1> [2018-05-05T20:41:37,927][INFO ][o.e.c.s.ClusterSettings ] [node_s0] updating [cluster.routing.rebalance.enable] from [none] to [all] 1> [2018-05-05T20:41:37,932][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testCreateSplitIndex]: cleaning up after test 1> [2018-05-05T20:41:37,944][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [source/ptLmt_0ZT-il5H1Rwd6k2A] deleting index 1> [2018-05-05T20:41:37,945][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [target/bxh3kPxPS8KpzQt53gOX9Q] deleting index 1> [2018-05-05T20:41:38,078][WARN ][o.e.i.IndicesService ] [node_s0] [source/ptLmt_0ZT-il5H1Rwd6k2A] failed to delete index 1> java.io.IOException: could not remove the following files (in the order of attempts): 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\MmHPJSThcp\0\ptLmt_0ZT-il5H1Rwd6k2A\0\index\_0.cfs: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\MmHPJSThcp\0\ptLmt_0ZT-il5H1Rwd6k2A\0\index\_0.cfs 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\MmHPJSThcp\0\ptLmt_0ZT-il5H1Rwd6k2A\0\index: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\MmHPJSThcp\0\ptLmt_0ZT-il5H1Rwd6k2A\0\index 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\MmHPJSThcp\0\ptLmt_0ZT-il5H1Rwd6k2A\0: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\MmHPJSThcp\0\ptLmt_0ZT-il5H1Rwd6k2A\0 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\MmHPJSThcp\0\ptLmt_0ZT-il5H1Rwd6k2A: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\MmHPJSThcp\0\ptLmt_0ZT-il5H1Rwd6k2A 1> at org.elasticsearch.core.internal.io.IOUtils.rm(IOUtils.java:187) ~[elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectoryUnderLock(NodeEnvironment.java:509) ~[main/:?] 1> at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectorySafe(NodeEnvironment.java:489) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.deleteIndexStoreIfDeletionAllowed(IndicesService.java:668) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.deleteIndexStore(IndicesService.java:655) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:561) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:284) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:218) ~[main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:496) ~[main/:?] 1> at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_162] 1> at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:493) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:480) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:431) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:161) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:38,111][WARN ][o.e.i.IndicesService ] [node_s0] [source/ptLmt_0ZT-il5H1Rwd6k2A] still pending deletes present for shards [[[source/ptLmt_0ZT-il5H1Rwd6k2A]], [[source/ptLmt_0ZT-il5H1Rwd6k2A]][0]] - retrying 1> [2018-05-05T20:41:38,143][WARN ][o.e.i.IndicesService ] [node_s0] [source/ptLmt_0ZT-il5H1Rwd6k2A] still pending deletes present for shards [[[source/ptLmt_0ZT-il5H1Rwd6k2A]]] - retrying 1> [2018-05-05T20:41:38,183][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [random_index_template] 1> [2018-05-05T20:41:38,205][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testCreateSplitIndex]: cleaned up after test 1> [2018-05-05T20:41:38,205][INFO ][o.e.a.a.i.c.SplitIndexIT ] [testCreateSplitIndex]: after test 1> [2018-05-05T20:41:38,232][INFO ][o.e.a.a.i.c.SplitIndexIT ] [testSplitFromOneToN]: before test 1> [2018-05-05T20:41:38,232][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testSplitFromOneToN]: setting up test 1> [2018-05-05T20:41:38,256][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [random_index_template] for index patterns [*] 1> [2018-05-05T20:41:38,268][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testSplitFromOneToN]: all set up test 1> [2018-05-05T20:41:38,268][INFO ][o.e.a.a.i.c.SplitIndexIT ] using custom data_path for index: [PujzCvists] 1> [2018-05-05T20:41:38,268][INFO ][o.e.a.a.i.c.SplitIndexIT ] use routing true use mixed routing true use nested false 1> [2018-05-05T20:41:38,271][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [source] creating index, cause [api], templates [random_index_template], shards [1]/[0], mappings [] 1> [2018-05-05T20:41:38,398][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][0]] ...]). 1> [2018-05-05T20:41:38,442][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [source/NdruUetVSNWkDkeHIZ45rw] create_mapping [t1] 1> [2018-05-05T20:41:38,611][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [first_split] creating index, cause [split_index], templates [], shards [5]/[0], mappings [] 1> [2018-05-05T20:41:38,689][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [first_split/hzB2fzCKRzuYU0OPpoTsHA] create_mapping [t1] 1> [2018-05-05T20:41:39,070][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[first_split][4]] ...]). 1> [2018-05-05T20:41:39,875][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [second_split] creating index, cause [split_index], templates [], shards [10]/[0], mappings [] 1> [2018-05-05T20:41:40,125][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [second_split/Tz4s7W-_RF-IOLbJ6VCUQA] create_mapping [t1] 1> [2018-05-05T20:41:40,770][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[second_split][8]] ...]). 1> [2018-05-05T20:41:41,227][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [1] for indices [second_split] 1> [2018-05-05T20:41:42,065][WARN ][o.e.i.s.Store ] [node_s0] [second_split][4] failed to build store metadata. checking segment info integrity (with commit [no]) 1> java.nio.file.NoSuchFileException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Tz4s7W-_RF-IOLbJ6VCUQA\4\index\_0.cfs 1> at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) ~[?:?] 1> at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115) ~[?:?] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.DisableFsyncFS.newFileChannel(DisableFsyncFS.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[?:1.8.0_162] 1> at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[?:1.8.0_162] 1> at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:912) ~[main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:878) [main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:797) [main/:?] 1> at org.elasticsearch.index.store.Store.getMetadata(Store.java:293) [main/:?] 1> at org.elasticsearch.index.store.Store.getMetadata(Store.java:258) [main/:?] 1> at org.elasticsearch.index.store.Store.cleanupAndVerify(Store.java:689) [main/:?] 1> at org.elasticsearch.indices.recovery.RecoveryTarget.cleanFiles(RecoveryTarget.java:440) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$CleanFilesRequestHandler.messageReceived(PeerRecoveryTargetService.java:556) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$CleanFilesRequestHandler.messageReceived(PeerRecoveryTargetService.java:550) [main/:?] 1> at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) [main/:?] 1> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [main/:?] 1> at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1603) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [main/:?] 1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:42,077][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[second_split][4]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [second_split][4]: Recovery failed from {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} into {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} (failed to clean after recovery) 1> at org.elasticsearch.indices.recovery.RecoveryTarget.cleanFiles(RecoveryTarget.java:468) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$CleanFilesRequestHandler.messageReceived(PeerRecoveryTargetService.java:556) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$CleanFilesRequestHandler.messageReceived(PeerRecoveryTargetService.java:550) [main/:?] 1> at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) [main/:?] 1> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [main/:?] 1> at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1603) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [main/:?] 1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: java.nio.file.NoSuchFileException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Tz4s7W-_RF-IOLbJ6VCUQA\4\index\_0.cfs 1> at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) ~[?:?] 1> at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115) ~[?:?] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.DisableFsyncFS.newFileChannel(DisableFsyncFS.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[?:1.8.0_162] 1> at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[?:1.8.0_162] 1> at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:912) ~[main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:878) ~[main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:797) ~[main/:?] 1> at org.elasticsearch.index.store.Store.getMetadata(Store.java:293) ~[main/:?] 1> at org.elasticsearch.index.store.Store.getMetadata(Store.java:258) ~[main/:?] 1> at org.elasticsearch.index.store.Store.cleanupAndVerify(Store.java:689) ~[main/:?] 1> at org.elasticsearch.indices.recovery.RecoveryTarget.cleanFiles(RecoveryTarget.java:440) [main/:?] 1> ... 10 more 1> [2018-05-05T20:41:42,080][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [second_split][4], node[YOWAmP1SSD2IFm9IQ_0seA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=-rD5dpI_Sk2JoCPSw57_vg], unassigned_info[[reason=REPLICA_ADDED], at[2018-05-05T08:41:41.227Z], delayed=false, allocation_status[no_attempt]], message [failed recovery], failure [RecoveryFailedException[[second_split][4]: Recovery failed from {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} into {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} (failed to clean after recovery)]; nested: NoSuchFileException[C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Tz4s7W-_RF-IOLbJ6VCUQA\4\index\_0.cfs]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [second_split][4]: Recovery failed from {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141} into {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139} (failed to clean after recovery) 1> at org.elasticsearch.indices.recovery.RecoveryTarget.cleanFiles(RecoveryTarget.java:468) ~[main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$CleanFilesRequestHandler.messageReceived(PeerRecoveryTargetService.java:556) ~[main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$CleanFilesRequestHandler.messageReceived(PeerRecoveryTargetService.java:550) ~[main/:?] 1> at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) ~[main/:?] 1> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[main/:?] 1> at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1603) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: java.nio.file.NoSuchFileException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Tz4s7W-_RF-IOLbJ6VCUQA\4\index\_0.cfs 1> at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) ~[?:?] 1> at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115) ~[?:?] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.DisableFsyncFS.newFileChannel(DisableFsyncFS.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[?:1.8.0_162] 1> at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[?:1.8.0_162] 1> at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:912) ~[main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:878) ~[main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:797) ~[main/:?] 1> at org.elasticsearch.index.store.Store.getMetadata(Store.java:293) ~[main/:?] 1> at org.elasticsearch.index.store.Store.getMetadata(Store.java:258) ~[main/:?] 1> at org.elasticsearch.index.store.Store.cleanupAndVerify(Store.java:689) ~[main/:?] 1> at org.elasticsearch.indices.recovery.RecoveryTarget.cleanFiles(RecoveryTarget.java:440) ~[main/:?] 1> ... 10 more 1> [2018-05-05T20:41:42,099][WARN ][o.e.i.s.TransportNodesListShardStoreMetaData] [node_s0] failed to build store metadata. checking segment info integrity (with commit [no]) 1> java.nio.file.NoSuchFileException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Tz4s7W-_RF-IOLbJ6VCUQA\4\index\_0.cfs 1> at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) ~[?:?] 1> at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:230) ~[?:?] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:240) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:240) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_162] 1> at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_162] 1> at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:912) ~[main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:878) [main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:797) [main/:?] 1> at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:464) [main/:?] 1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:150) [main/:?] 1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:109) [main/:?] 1> at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61) [main/:?] 1> at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) [main/:?] 1> at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:260) [main/:?] 1> at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:256) [main/:?] 1> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [main/:?] 1> at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:656) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [main/:?] 1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:42,101][INFO ][o.e.i.s.TransportNodesListShardStoreMetaData] [node_s0] Failed to open / find files while reading metadata snapshot 1> [2018-05-05T20:41:42,121][WARN ][o.e.i.s.Store ] [node_s0] [second_split][4] failed to build store metadata. checking segment info integrity (with commit [no]) 1> java.nio.file.NoSuchFileException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Tz4s7W-_RF-IOLbJ6VCUQA\4\index\_0.cfs 1> at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) ~[?:?] 1> at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115) ~[?:?] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.DisableFsyncFS.newFileChannel(DisableFsyncFS.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[?:1.8.0_162] 1> at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[?:1.8.0_162] 1> at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:912) ~[main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:878) [main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:797) [main/:?] 1> at org.elasticsearch.index.store.Store.getMetadata(Store.java:293) [main/:?] 1> at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1103) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:294) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:315) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:178) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:80) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:623) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [main/:?] 1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:42,123][WARN ][o.e.i.r.PeerRecoveryTargetService] [node_s0] error while listing local files, recovering as if there are none 1> java.nio.file.NoSuchFileException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\Tz4s7W-_RF-IOLbJ6VCUQA\4\index\_0.cfs 1> at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) ~[?:?] 1> at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115) ~[?:?] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.DisableFsyncFS.newFileChannel(DisableFsyncFS.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[?:1.8.0_162] 1> at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[?:1.8.0_162] 1> at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:912) ~[main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:878) ~[main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:797) ~[main/:?] 1> at org.elasticsearch.index.store.Store.getMetadata(Store.java:293) ~[main/:?] 1> at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1103) ~[main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStoreMetadataSnapshot(PeerRecoveryTargetService.java:294) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:315) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:178) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:80) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:623) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [main/:?] 1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:42,306][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[second_split][4]] ...]). 1> [2018-05-05T20:41:43,396][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [first_split/hzB2fzCKRzuYU0OPpoTsHA] deleting index 1> [2018-05-05T20:41:43,396][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [source/NdruUetVSNWkDkeHIZ45rw] deleting index 1> [2018-05-05T20:41:43,396][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [second_split/Tz4s7W-_RF-IOLbJ6VCUQA] deleting index 1> [2018-05-05T20:41:44,185][INFO ][o.e.a.a.i.c.SplitIndexIT ] use routing false use mixed routing false use nested true 1> [2018-05-05T20:41:44,188][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [source] creating index, cause [api], templates [random_index_template], shards [1]/[1], mappings [t1] 1> [2018-05-05T20:41:44,319][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [source/cC7P5_OtRLS1RQHmAuWGAA] update_mapping [t1] 1> [2018-05-05T20:41:44,799][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][0]] ...]). 1> [2018-05-05T20:41:44,926][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [first_split] creating index, cause [split_index], templates [], shards [2]/[0], mappings [] 1> [2018-05-05T20:41:45,050][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [first_split/HTC32XIKSvyqzOd2tRqBng] create_mapping [t1] 1> [2018-05-05T20:41:45,169][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[first_split][1]] ...]). 1> [2018-05-05T20:41:45,649][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [second_split] creating index, cause [split_index], templates [], shards [4]/[0], mappings [] 1> [2018-05-05T20:41:45,716][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [second_split/AUgsRDvNTEyNiUs-DlfhTA] create_mapping [t1] 1> [2018-05-05T20:41:45,901][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[second_split][1], [second_split][2], [second_split][0]] ...]). 1> [2018-05-05T20:41:45,921][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [1] for indices [second_split] 1> [2018-05-05T20:41:46,328][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[second_split][3], [second_split][1], [second_split][0]] ...]). 1> [2018-05-05T20:41:47,410][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testSplitFromOneToN]: cleaning up after test 1> [2018-05-05T20:41:47,444][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [source/cC7P5_OtRLS1RQHmAuWGAA] deleting index 1> [2018-05-05T20:41:47,444][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [first_split/HTC32XIKSvyqzOd2tRqBng] deleting index 1> [2018-05-05T20:41:47,444][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [second_split/AUgsRDvNTEyNiUs-DlfhTA] deleting index 1> [2018-05-05T20:41:47,858][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [random_index_template] 1> [2018-05-05T20:41:47,868][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testSplitFromOneToN]: cleaned up after test 1> [2018-05-05T20:41:47,868][INFO ][o.e.a.a.i.c.SplitIndexIT ] [testSplitFromOneToN]: after test 1> [2018-05-05T20:41:47,906][INFO ][o.e.a.a.i.c.SplitIndexIT ] [testCreateSplitWithIndexSort]: before test 1> [2018-05-05T20:41:47,907][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testCreateSplitWithIndexSort]: setting up test 1> [2018-05-05T20:41:47,930][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [random_index_template] for index patterns [*] 1> [2018-05-05T20:41:47,954][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testCreateSplitWithIndexSort]: all set up test 1> [2018-05-05T20:41:47,957][INFO ][o.e.a.a.i.c.SplitIndexIT ] using custom data_path for index: [MpztASNIMf] 1> [2018-05-05T20:41:47,957][INFO ][o.e.a.a.i.c.SplitIndexIT ] using custom data_path for index: [YlAKUxPjnS] 1> [2018-05-05T20:41:47,961][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [source] creating index, cause [api], templates [random_index_template], shards [2]/[0], mappings [type] 1> [2018-05-05T20:41:48,088][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][0]] ...]). 1> [2018-05-05T20:41:48,104][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [source/N36JFkQQTEGhTzMhSearvg] update_mapping [type] 1> [2018-05-05T20:41:48,534][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [target] creating index, cause [split_index], templates [], shards [4]/[0], mappings [] 1> [2018-05-05T20:41:48,699][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [target/aeV2i-PiSuu5aKqs0Egtyw] create_mapping [type] 1> [2018-05-05T20:41:48,961][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[target][1]] ...]). 1> [2018-05-05T20:41:48,982][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [target/aeV2i-PiSuu5aKqs0Egtyw] update_mapping [type] 1> [2018-05-05T20:41:49,212][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testCreateSplitWithIndexSort]: cleaning up after test 1> [2018-05-05T20:41:49,222][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [target/aeV2i-PiSuu5aKqs0Egtyw] deleting index 1> [2018-05-05T20:41:49,222][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [source/N36JFkQQTEGhTzMhSearvg] deleting index 1> [2018-05-05T20:41:49,250][WARN ][o.e.i.IndicesService ] [node_s1] [source/N36JFkQQTEGhTzMhSearvg] failed to delete index 1> java.io.IOException: could not remove the following files (in the order of attempts): 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\YlAKUxPjnS\1\N36JFkQQTEGhTzMhSearvg\1\index\_0.cfs: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\YlAKUxPjnS\1\N36JFkQQTEGhTzMhSearvg\1\index\_0.cfs 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\YlAKUxPjnS\1\N36JFkQQTEGhTzMhSearvg\1\index: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\YlAKUxPjnS\1\N36JFkQQTEGhTzMhSearvg\1\index 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\YlAKUxPjnS\1\N36JFkQQTEGhTzMhSearvg\1: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\YlAKUxPjnS\1\N36JFkQQTEGhTzMhSearvg\1 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\YlAKUxPjnS\1\N36JFkQQTEGhTzMhSearvg: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\YlAKUxPjnS\1\N36JFkQQTEGhTzMhSearvg 1> at org.elasticsearch.core.internal.io.IOUtils.rm(IOUtils.java:187) ~[elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectoryUnderLock(NodeEnvironment.java:509) ~[main/:?] 1> at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectorySafe(NodeEnvironment.java:489) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.deleteIndexStoreIfDeletionAllowed(IndicesService.java:668) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.deleteIndexStore(IndicesService.java:655) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:561) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:284) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:218) ~[main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:496) ~[main/:?] 1> at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_162] 1> at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:493) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:480) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:431) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:161) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:49,275][WARN ][o.e.i.IndicesService ] [node_s1] [source/N36JFkQQTEGhTzMhSearvg] still pending deletes present for shards [[[source/N36JFkQQTEGhTzMhSearvg]], [[source/N36JFkQQTEGhTzMhSearvg]][1]] - retrying 1> [2018-05-05T20:41:49,341][WARN ][o.e.i.IndicesService ] [node_s0] [source/N36JFkQQTEGhTzMhSearvg] failed to delete index 1> java.io.IOException: could not remove the following files (in the order of attempts): 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\YlAKUxPjnS\0\N36JFkQQTEGhTzMhSearvg\0\index\_0.cfs: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\YlAKUxPjnS\0\N36JFkQQTEGhTzMhSearvg\0\index\_0.cfs 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\YlAKUxPjnS\0\N36JFkQQTEGhTzMhSearvg\0\index: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\YlAKUxPjnS\0\N36JFkQQTEGhTzMhSearvg\0\index 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\YlAKUxPjnS\0\N36JFkQQTEGhTzMhSearvg\0: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\YlAKUxPjnS\0\N36JFkQQTEGhTzMhSearvg\0 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\YlAKUxPjnS\0\N36JFkQQTEGhTzMhSearvg: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001\tempDir-002\custom\YlAKUxPjnS\0\N36JFkQQTEGhTzMhSearvg 1> at org.elasticsearch.core.internal.io.IOUtils.rm(IOUtils.java:187) ~[elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectoryUnderLock(NodeEnvironment.java:509) ~[main/:?] 1> at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectorySafe(NodeEnvironment.java:489) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.deleteIndexStoreIfDeletionAllowed(IndicesService.java:668) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.deleteIndexStore(IndicesService.java:655) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:561) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:284) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:218) ~[main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:496) ~[main/:?] 1> at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_162] 1> at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:493) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:480) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:431) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:161) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T20:41:49,364][WARN ][o.e.i.IndicesService ] [node_s0] [source/N36JFkQQTEGhTzMhSearvg] still pending deletes present for shards [[[source/N36JFkQQTEGhTzMhSearvg]], [[source/N36JFkQQTEGhTzMhSearvg]][0]] - retrying 1> [2018-05-05T20:41:49,428][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [random_index_template] 1> [2018-05-05T20:41:49,448][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testCreateSplitWithIndexSort]: cleaned up after test 1> [2018-05-05T20:41:49,448][INFO ][o.e.a.a.i.c.SplitIndexIT ] [testCreateSplitWithIndexSort]: after test 1> [2018-05-05T20:41:49,520][INFO ][o.e.a.a.i.c.SplitIndexIT ] [testCreateSplitIndexToN]: before test 1> [2018-05-05T20:41:49,520][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testCreateSplitIndexToN]: setting up test 1> [2018-05-05T20:41:49,695][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [random_index_template] for index patterns [*] 1> [2018-05-05T20:41:49,737][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testCreateSplitIndexToN]: all set up test 1> [2018-05-05T20:41:49,737][INFO ][o.e.a.a.i.c.SplitIndexIT ] use routing true use mixed routing true use nested true 1> [2018-05-05T20:41:49,743][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [source] creating index, cause [api], templates [random_index_template], shards [1]/[1], mappings [t1] 1> [2018-05-05T20:41:50,060][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [source/DnB-yL-3Snm_09RY3AMUWw] update_mapping [t1] 1> [2018-05-05T20:41:50,226][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][0]] ...]). 1> [2018-05-05T20:41:50,247][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [first_split] creating index, cause [split_index], templates [], shards [2]/[0], mappings [] 1> [2018-05-05T20:41:50,427][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [first_split/sVS2JF3_T7uldJXm6V482g] create_mapping [t1] 1> [2018-05-05T20:41:50,644][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[first_split][0]] ...]). 1> [2018-05-05T20:41:51,038][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [second_split] creating index, cause [split_index], templates [], shards [4]/[0], mappings [] 1> [2018-05-05T20:41:51,125][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [second_split/E2FyhbOET_Oynu6UfdcnHQ] create_mapping [t1] 1> [2018-05-05T20:41:51,332][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[second_split][2]] ...]). 1> [2018-05-05T20:41:51,389][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [1] for indices [second_split] 1> [2018-05-05T20:41:51,593][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[second_split][3], [second_split][0], [second_split][2]] ...]). 1> [2018-05-05T20:41:52,184][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testCreateSplitIndexToN]: cleaning up after test 1> [2018-05-05T20:41:52,195][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [source/DnB-yL-3Snm_09RY3AMUWw] deleting index 1> [2018-05-05T20:41:52,195][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [second_split/E2FyhbOET_Oynu6UfdcnHQ] deleting index 1> [2018-05-05T20:41:52,195][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [first_split/sVS2JF3_T7uldJXm6V482g] deleting index 1> [2018-05-05T20:41:52,487][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [random_index_template] 1> [2018-05-05T20:41:52,499][INFO ][o.e.a.a.i.c.SplitIndexIT ] [SplitIndexIT#testCreateSplitIndexToN]: cleaned up after test 1> [2018-05-05T20:41:52,500][INFO ][o.e.a.a.i.c.SplitIndexIT ] [testCreateSplitIndexToN]: after test 1> [2018-05-05T20:41:52,505][INFO ][o.e.n.Node ] [node_s0] stopping ... 1> [2018-05-05T20:41:52,506][INFO ][o.e.t.d.TestZenDiscovery ] [node_s1] master_left [{node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}], reason [shut_down] 1> [2018-05-05T20:41:52,506][WARN ][o.e.t.d.TestZenDiscovery ] [node_s1] master left (reason = shut_down), current nodes: nodes: 1> {node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}, master 1> {node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}, local 1> 1> [2018-05-05T20:41:52,506][INFO ][o.e.c.s.ClusterApplierService] [node_s1] master node changed {previous [{node_s0}{YOWAmP1SSD2IFm9IQ_0seA}{ZxHi1icuSEy8CkWVxxuChA}{127.0.0.1}{127.0.0.1:54139}], current []}, reason: master left (reason = shut_down) 1> [2018-05-05T20:41:52,506][INFO ][o.e.t.d.MockZenPing ] [node_s1] pinging using mock zen ping 1> [2018-05-05T20:41:52,509][INFO ][o.e.n.Node ] [node_s0] stopped 1> [2018-05-05T20:41:52,509][INFO ][o.e.n.Node ] [node_s0] closing ... 1> [2018-05-05T20:41:52,512][INFO ][o.e.n.Node ] [node_s0] closed 1> [2018-05-05T20:41:52,522][INFO ][o.e.n.Node ] [node_s1] stopping ... 1> [2018-05-05T20:41:52,522][WARN ][o.e.t.d.TestZenDiscovery ] [node_s1] not enough master nodes discovered during pinging (found [[Candidate{node={node_s1}{mIq8KinYToaJSuyTqY1_hQ}{AGMcJC0yTymYVA12WHphHQ}{127.0.0.1}{127.0.0.1:54141}, clusterStateVersion=315}]], but needed [2]), pinging again 1> [2018-05-05T20:41:52,525][INFO ][o.e.n.Node ] [node_s1] stopped 1> [2018-05-05T20:41:52,525][INFO ][o.e.n.Node ] [node_s1] closing ... 1> [2018-05-05T20:41:52,528][INFO ][o.e.n.Node ] [node_s1] closed 2> NOTE: leaving temporary files on disk at: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J1\temp\org.elasticsearch.action.admin.indices.create.SplitIndexIT_734B52BB31887C24-001 2> NOTE: test params are: codec=Asserting(Lucene70): {_routing=PostingsFormat(name=Asserting), nested1.n_field1=PostingsFormat(name=Asserting), nested1.n_field2=PostingsFormat(name=Asserting), nested1.n_field2.keyword=PostingsFormat(name=Asserting), nested1.n_field1.keyword=PostingsFormat(name=Asserting), f=PostingsFormat(name=Asserting), foo=PostingsFormat(name=Asserting), _type=PostingsFormat(name=Asserting), f.keyword=PostingsFormat(name=Asserting), foo.keyword=PostingsFormat(name=Asserting), _field_names=PostingsFormat(name=Asserting), _id=PostingsFormat(name=Asserting), id=PostingsFormat(name=Asserting)}, docValues:{_seq_no=DocValuesFormat(name=Asserting), nested1.n_field2.keyword=DocValuesFormat(name=Asserting), nested1.n_field1.keyword=DocValuesFormat(name=Lucene70), i=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Lucene70), _primary_term=DocValuesFormat(name=Lucene70), _version=DocValuesFormat(name=Asserting), f.keyword=DocValuesFormat(name=Lucene70), foo.keyword=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1183, maxMBSortInHeap=6.481888729005478, sim=RandomSimilarity(queryNorm=false): {}, locale=sq, timezone=Antarctica/McMurdo 2> NOTE: Windows Server 2016 10.0 amd64/Oracle Corporation 1.8.0_162 (64-bit)/cpus=4,threads=1,free=352899144,total=526909440 2> NOTE: All tests run in this JVM: [SearchFieldsIT, SimilarityIT, GeoDistanceIT, FieldDataLoadingIT, CorruptedTranslogIT, BroadcastActionsIT, ClusterDisruptionIT, MultiTermVectorsIT, SimpleNestedIT, RareClusterStateIT, SearchRedStateIndexIT, ListenerActionIT, ScriptedMetricIT, FsBlobStoreRepositoryIT, SearchSliceIT, ConcurrentDynamicTemplateIT, QueryRescorerIT, RecoveryWhileUnderLoadIT, IndicesRequestCacheIT, SplitIndexIT] Completed [53/296] on J1 in 50.65s, 5 tests, 1 failure <<< FAILURES! Suite: org.elasticsearch.action.support.WaitActiveShardCountIT Completed [54/296] on J1 in 3.62s, 1 test Suite: org.elasticsearch.search.simple.SimpleSearchIT Completed [55/296] on J0 in 25.13s, 21 tests Suite: org.elasticsearch.search.geo.GeoShapeIntegrationIT Completed [56/296] on J1 in 5.53s, 2 tests Suite: org.elasticsearch.search.query.SearchQueryIT Completed [57/296] on J2 in 41.45s, 51 tests Suite: org.elasticsearch.search.functionscore.FunctionScoreIT Completed [58/296] on J2 in 2.92s, 5 tests Suite: org.elasticsearch.indices.recovery.IndexRecoveryIT Completed [59/296] on J1 in 16.12s, 7 tests Suite: org.elasticsearch.action.admin.cluster.node.tasks.TasksIT Completed [60/296] on J2 in 12.21s, 24 tests Suite: org.elasticsearch.indices.flush.FlushIT Completed [61/296] on J0 in 26.56s, 6 tests Suite: org.elasticsearch.indices.settings.UpdateSettingsIT Completed [62/296] on J1 in 10.07s, 9 tests Suite: org.elasticsearch.search.functionscore.FunctionScorePluginIT Completed [63/296] on J0 in 1.92s, 1 test Suite: org.elasticsearch.search.aggregations.metrics.ValueCountIT Completed [64/296] on J1 in 6.05s, 11 tests Suite: org.elasticsearch.aliases.IndexAliasesIT Completed [65/296] on J2 in 15.45s, 26 tests HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:42:39, stalled for 11.4s at: FullRollingRestartIT.testFullRollingRestart HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:42:45, stalled for 11.6s at: StressSearchServiceReaperIT.testStressReaper HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:42:45, stalled for 10.1s at: IndexingMasterFailoverIT.testMasterFailoverDuringIndexingWithMappingChanges Suite: org.elasticsearch.search.StressSearchServiceReaperIT Completed [66/296] on J1 in 14.17s, 1 test HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:42:49, stalled for 21.4s at: FullRollingRestartIT.testFullRollingRestart Suite: org.elasticsearch.cluster.allocation.FilteringAllocationIT Completed [67/296] on J1 in 3.99s, 4 tests HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:42:55, stalled for 20.1s at: IndexingMasterFailoverIT.testMasterFailoverDuringIndexingWithMappingChanges Suite: org.elasticsearch.indices.DateMathIndexExpressionsIntegrationIT Completed [68/296] on J1 in 3.95s, 3 tests Suite: org.elasticsearch.action.support.master.IndexingMasterFailoverIT Completed [69/296] on J2 in 21.52s, 1 test Suite: org.elasticsearch.recovery.FullRollingRestartIT Completed [70/296] on J0 in 29.36s, 2 tests Suite: org.elasticsearch.update.UpdateNoopIT Completed [71/296] on J1 in 3.61s, 6 tests Suite: org.elasticsearch.cluster.allocation.AwarenessAllocationIT Completed [72/296] on J2 in 9.20s, 3 tests Suite: org.elasticsearch.persistent.decider.EnableAssignmentDeciderIT Completed [73/296] on J2 in 4.61s, 1 test Suite: org.elasticsearch.indices.mapping.DedicatedMasterGetFieldMappingIT Completed [74/296] on J2 in 7.62s, 5 tests Suite: org.elasticsearch.action.bulk.BulkWithUpdatesIT Completed [75/296] on J1 in 19.80s, 11 tests Suite: org.elasticsearch.indices.IndicesOptionsIntegrationIT Completed [76/296] on J0 in 27.20s, 16 tests HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:43:30, stalled for 11.4s at: IndexActionIT.testAutoGenerateIdNoDuplicates Suite: org.elasticsearch.indices.exists.indices.IndicesExistsIT Completed [77/296] on J1 in 12.13s, 2 tests Suite: org.elasticsearch.indexing.IndexActionIT Completed [78/296] on J2 in 20.24s, 9 tests Suite: org.elasticsearch.cluster.settings.SettingsFilteringIT Completed [79/296] on J2 in 0.39s, 2 tests HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:43:40, stalled for 11.6s at: ShrinkIndexIT.testCreateShrinkIndexToN Suite: org.elasticsearch.index.translog.TruncateTranslogIT Completed [80/296] on J1 in 10.73s, 2 tests Suite: org.elasticsearch.action.admin.indices.refresh.RefreshBlocksIT Completed [81/296] on J1 in 0.53s, 1 test HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:43:50, stalled for 21.6s at: ShrinkIndexIT.testCreateShrinkIndexToN HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:43:54, stalled for 11.0s at: PercentilesBucketIT (suite) Suite: org.elasticsearch.search.aggregations.pipeline.PercentilesBucketIT Completed [82/296] on J1 in 13.32s, 12 tests HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:44:00, stalled for 31.6s at: ShrinkIndexIT.testCreateShrinkIndexToN HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:44:04, stalled for 10.7s at: CreateIndexIT.testCreateIndexWithMetadataBlocks HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:44:08, stalled for 11.7s at: GlobalCheckpointSyncIT.testBackgroundGlobalCheckpointSync Suite: org.elasticsearch.action.admin.indices.create.ShrinkIndexIT 1> [2018-05-05T08:43:24,937][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [testCreateShrinkIndex]: before test 1> [2018-05-05T08:43:24,939][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testCreateShrinkIndex]: setting up test 1> [2018-05-05T08:43:24,942][INFO ][o.e.t.InternalTestCluster] Setup InternalTestCluster [SUITE-CHILD_VM=[0]-CLUSTER_SEED=[-8100806132142761887]-HASH=[56C379D2F6B4FC]-cluster] with seed [8F9427BE23531861] using [0] dedicated masters, [2] (data) nodes and [1] coord only nodes (min_master_nodes are [auto-managed]) 1> [2018-05-05T08:43:24,946][INFO ][o.e.n.Node ] [node_s0] initializing ... 1> [2018-05-05T08:43:24,963][INFO ][o.e.e.NodeEnvironment ] [node_s0] using [1] data paths, mounts [[Windows (C:)]], net usable_space [379.2gb], net total_space [499.9gb], types [NTFS] 1> [2018-05-05T08:43:24,963][INFO ][o.e.e.NodeEnvironment ] [node_s0] heap size [491mb], compressed ordinary object pointers [true] 1> [2018-05-05T08:43:24,965][INFO ][o.e.n.Node ] [node_s0] node name [node_s0], node ID [_jxN82x-QuiKqw6LSus-lg] 1> [2018-05-05T08:43:24,965][INFO ][o.e.n.Node ] [node_s0] version[7.0.0-alpha1-SNAPSHOT], pid[6640], build[unknown/unknown/Unknown/Unknown], OS[Windows Server 2016/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_162/25.162-b12] 1> [2018-05-05T08:43:24,966][INFO ][o.e.n.Node ] [node_s0] JVM arguments [-ea, -esa, -Xmx512m, -Xms512m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\heapdump, -Dtests.prefix=tests, -Dtests.seed=734B52BB31887C24, -Dtests.security.manager=true, -Djava.io.tmpdir=./temp, -Dtests.gradle=true, -Dtests.artifact=server, -Djava.awt.headless=true, -Dtests.task=:server:integTest, -Djna.nosys=true, -Dtests.logger.level=WARN, -Djunit4.childvm.cwd=C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0, -Djunit4.childvm.id=0, -Djunit4.childvm.count=3] 1> [2018-05-05T08:43:24,966][WARN ][o.e.n.Node ] [node_s0] version [7.0.0-alpha1-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production 1> [2018-05-05T08:43:24,968][INFO ][o.e.p.PluginsService ] [node_s0] no modules loaded 1> [2018-05-05T08:43:24,968][INFO ][o.e.p.PluginsService ] [node_s0] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] 1> [2018-05-05T08:43:24,968][INFO ][o.e.p.PluginsService ] [node_s0] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] 1> [2018-05-05T08:43:24,968][INFO ][o.e.p.PluginsService ] [node_s0] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] 1> [2018-05-05T08:43:24,968][INFO ][o.e.p.PluginsService ] [node_s0] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin] 1> [2018-05-05T08:43:24,968][INFO ][o.e.p.PluginsService ] [node_s0] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] 1> [2018-05-05T08:43:25,007][INFO ][o.e.d.DiscoveryModule ] [node_s0] using discovery type [test-zen] 1> [2018-05-05T08:43:25,036][INFO ][o.e.n.Node ] [node_s0] initialized 1> [2018-05-05T08:43:25,043][INFO ][o.e.n.Node ] [node_s1] initializing ... 1> [2018-05-05T08:43:25,089][INFO ][o.e.e.NodeEnvironment ] [node_s1] using [1] data paths, mounts [[Windows (C:)]], net usable_space [379.2gb], net total_space [499.9gb], types [NTFS] 1> [2018-05-05T08:43:25,089][INFO ][o.e.e.NodeEnvironment ] [node_s1] heap size [491mb], compressed ordinary object pointers [true] 1> [2018-05-05T08:43:25,100][INFO ][o.e.n.Node ] [node_s1] node name [node_s1], node ID [g9obCv-7RBCJawEoUGOH2Q] 1> [2018-05-05T08:43:25,100][INFO ][o.e.n.Node ] [node_s1] version[7.0.0-alpha1-SNAPSHOT], pid[6640], build[unknown/unknown/Unknown/Unknown], OS[Windows Server 2016/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_162/25.162-b12] 1> [2018-05-05T08:43:25,100][INFO ][o.e.n.Node ] [node_s1] JVM arguments [-ea, -esa, -Xmx512m, -Xms512m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\heapdump, -Dtests.prefix=tests, -Dtests.seed=734B52BB31887C24, -Dtests.security.manager=true, -Djava.io.tmpdir=./temp, -Dtests.gradle=true, -Dtests.artifact=server, -Djava.awt.headless=true, -Dtests.task=:server:integTest, -Djna.nosys=true, -Dtests.logger.level=WARN, -Djunit4.childvm.cwd=C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0, -Djunit4.childvm.id=0, -Djunit4.childvm.count=3] 1> [2018-05-05T08:43:25,100][WARN ][o.e.n.Node ] [node_s1] version [7.0.0-alpha1-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production 1> [2018-05-05T08:43:25,101][INFO ][o.e.p.PluginsService ] [node_s1] no modules loaded 1> [2018-05-05T08:43:25,101][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] 1> [2018-05-05T08:43:25,101][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] 1> [2018-05-05T08:43:25,101][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] 1> [2018-05-05T08:43:25,101][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin] 1> [2018-05-05T08:43:25,101][INFO ][o.e.p.PluginsService ] [node_s1] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] 1> [2018-05-05T08:43:25,128][INFO ][o.e.d.DiscoveryModule ] [node_s1] using discovery type [test-zen] 1> [2018-05-05T08:43:25,153][INFO ][o.e.n.Node ] [node_s1] initialized 1> [2018-05-05T08:43:25,156][INFO ][o.e.n.Node ] [node_sc2] initializing ... 1> [2018-05-05T08:43:25,170][INFO ][o.e.e.NodeEnvironment ] [node_sc2] using [1] data paths, mounts [[Windows (C:)]], net usable_space [379.2gb], net total_space [499.9gb], types [NTFS] 1> [2018-05-05T08:43:25,170][INFO ][o.e.e.NodeEnvironment ] [node_sc2] heap size [491mb], compressed ordinary object pointers [true] 1> [2018-05-05T08:43:25,173][INFO ][o.e.n.Node ] [node_sc2] node name [node_sc2], node ID [9vx9IZB3Tz2BJpMfrMcqqA] 1> [2018-05-05T08:43:25,173][INFO ][o.e.n.Node ] [node_sc2] version[7.0.0-alpha1-SNAPSHOT], pid[6640], build[unknown/unknown/Unknown/Unknown], OS[Windows Server 2016/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_162/25.162-b12] 1> [2018-05-05T08:43:25,173][INFO ][o.e.n.Node ] [node_sc2] JVM arguments [-ea, -esa, -Xmx512m, -Xms512m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\heapdump, -Dtests.prefix=tests, -Dtests.seed=734B52BB31887C24, -Dtests.security.manager=true, -Djava.io.tmpdir=./temp, -Dtests.gradle=true, -Dtests.artifact=server, -Djava.awt.headless=true, -Dtests.task=:server:integTest, -Djna.nosys=true, -Dtests.logger.level=WARN, -Djunit4.childvm.cwd=C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0, -Djunit4.childvm.id=0, -Djunit4.childvm.count=3] 1> [2018-05-05T08:43:25,173][WARN ][o.e.n.Node ] [node_sc2] version [7.0.0-alpha1-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production 1> [2018-05-05T08:43:25,173][INFO ][o.e.p.PluginsService ] [node_sc2] no modules loaded 1> [2018-05-05T08:43:25,174][INFO ][o.e.p.PluginsService ] [node_sc2] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin] 1> [2018-05-05T08:43:25,174][INFO ][o.e.p.PluginsService ] [node_sc2] loaded plugin [org.elasticsearch.test.InternalSettingsPlugin] 1> [2018-05-05T08:43:25,174][INFO ][o.e.p.PluginsService ] [node_sc2] loaded plugin [org.elasticsearch.test.MockHttpTransport$TestPlugin] 1> [2018-05-05T08:43:25,174][INFO ][o.e.p.PluginsService ] [node_sc2] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin] 1> [2018-05-05T08:43:25,174][INFO ][o.e.p.PluginsService ] [node_sc2] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] 1> [2018-05-05T08:43:25,193][INFO ][o.e.d.DiscoveryModule ] [node_sc2] using discovery type [test-zen] 1> [2018-05-05T08:43:25,214][INFO ][o.e.n.Node ] [node_sc2] initialized 1> [2018-05-05T08:43:25,218][INFO ][o.e.n.Node ] [node_s1] starting ... 1> [2018-05-05T08:43:25,218][INFO ][o.e.n.Node ] [node_sc2] starting ... 1> [2018-05-05T08:43:25,239][INFO ][o.e.n.Node ] [node_s0] starting ... 1> [2018-05-05T08:43:25,338][INFO ][o.e.t.TransportService ] [node_s0] publish_address {127.0.0.1:60041}, bound_addresses {127.0.0.1:60041}, {[::1]:60042} 1> [2018-05-05T08:43:25,340][INFO ][o.e.t.TransportService ] [node_s1] publish_address {127.0.0.1:60043}, bound_addresses {127.0.0.1:60043}, {[::1]:60044} 1> [2018-05-05T08:43:25,341][INFO ][o.e.t.d.MockZenPing ] [node_s0] pinging using mock zen ping 1> [2018-05-05T08:43:25,345][INFO ][o.e.t.d.MockZenPing ] [node_s1] pinging using mock zen ping 1> [2018-05-05T08:43:25,368][INFO ][o.e.t.TransportService ] [node_sc2] publish_address {127.0.0.1:60045}, bound_addresses {127.0.0.1:60045}, {[::1]:60046} 1> [2018-05-05T08:43:25,379][INFO ][o.e.t.d.MockZenPing ] [node_sc2] pinging using mock zen ping 1> [2018-05-05T08:43:25,423][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-elected-as-master ([1] nodes joined)[, ], reason: master node changed {previous [], current [{node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041}]}, added {{node_s1}{g9obCv-7RBCJawEoUGOH2Q}{DOOjJOB4Sae-PCzpbBgazA}{127.0.0.1}{127.0.0.1:60043},} 1> [2018-05-05T08:43:25,425][INFO ][o.e.c.s.ClusterApplierService] [node_s1] master node changed {previous [], current [{node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041}]}, added {{node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041},}, reason: apply cluster state (from master [master {node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041} committed version [1]]) 1> [2018-05-05T08:43:25,428][INFO ][o.e.n.Node ] [node_s1] started 1> [2018-05-05T08:43:25,428][INFO ][o.e.c.s.ClusterApplierService] [node_s0] master node changed {previous [], current [{node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041}]}, added {{node_s1}{g9obCv-7RBCJawEoUGOH2Q}{DOOjJOB4Sae-PCzpbBgazA}{127.0.0.1}{127.0.0.1:60043},}, reason: apply cluster state (from master [master {node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041} committed version [1] source [zen-disco-elected-as-master ([1] nodes joined)[, ]]]) 1> [2018-05-05T08:43:25,432][INFO ][o.e.n.Node ] [node_s0] started 1> [2018-05-05T08:43:25,433][INFO ][o.e.c.s.MasterService ] [node_s0] zen-disco-node-join, reason: added {{node_sc2}{9vx9IZB3Tz2BJpMfrMcqqA}{nb6v1D7YSUmzgPbW4eRV4w}{127.0.0.1}{127.0.0.1:60045},} 1> [2018-05-05T08:43:25,435][INFO ][o.e.c.s.ClusterApplierService] [node_sc2] master node changed {previous [], current [{node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041}]}, added {{node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041},{node_s1}{g9obCv-7RBCJawEoUGOH2Q}{DOOjJOB4Sae-PCzpbBgazA}{127.0.0.1}{127.0.0.1:60043},}, reason: apply cluster state (from master [master {node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041} committed version [2]]) 1> [2018-05-05T08:43:25,435][INFO ][o.e.c.s.ClusterApplierService] [node_s1] added {{node_sc2}{9vx9IZB3Tz2BJpMfrMcqqA}{nb6v1D7YSUmzgPbW4eRV4w}{127.0.0.1}{127.0.0.1:60045},}, reason: apply cluster state (from master [master {node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041} committed version [2]]) 1> [2018-05-05T08:43:25,449][INFO ][o.e.n.Node ] [node_sc2] started 1> [2018-05-05T08:43:25,450][INFO ][o.e.p.PluginsService ] [transport_client_node_s0] no modules loaded 1> [2018-05-05T08:43:25,450][INFO ][o.e.p.PluginsService ] [transport_client_node_s0] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] 1> [2018-05-05T08:43:25,451][INFO ][o.e.c.s.ClusterApplierService] [node_s0] added {{node_sc2}{9vx9IZB3Tz2BJpMfrMcqqA}{nb6v1D7YSUmzgPbW4eRV4w}{127.0.0.1}{127.0.0.1:60045},}, reason: apply cluster state (from master [master {node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041} committed version [2] source [zen-disco-node-join]]) 1> [2018-05-05T08:43:25,474][INFO ][o.e.p.PluginsService ] [transport_client_node_sc2] no modules loaded 1> [2018-05-05T08:43:25,474][INFO ][o.e.p.PluginsService ] [transport_client_node_sc2] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] 1> [2018-05-05T08:43:25,505][INFO ][o.e.g.GatewayService ] [node_s0] recovered [0] indices into cluster_state 1> [2018-05-05T08:43:25,508][INFO ][o.e.p.PluginsService ] [transport_client_node_s1] no modules loaded 1> [2018-05-05T08:43:25,508][INFO ][o.e.p.PluginsService ] [transport_client_node_s1] loaded plugin [org.elasticsearch.transport.nio.MockNioTransportPlugin] 1> [2018-05-05T08:43:25,526][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [random_index_template] for index patterns [*] 1> [2018-05-05T08:43:25,544][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testCreateShrinkIndex]: all set up test 1> [2018-05-05T08:43:25,548][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [source] creating index, cause [api], templates [random_index_template], shards [4]/[0], mappings [] 1> [2018-05-05T08:43:25,776][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][3]] ...]). 1> [2018-05-05T08:43:25,816][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [source/ZYmXwyQbSPCRp1ZZtc33OA] create_mapping [type] 1> [2018-05-05T08:43:26,402][INFO ][o.e.c.s.ClusterSettings ] [node_sc2] updating [cluster.routing.rebalance.enable] from [all] to [none] 1> [2018-05-05T08:43:26,422][INFO ][o.e.c.s.ClusterSettings ] [node_s1] updating [cluster.routing.rebalance.enable] from [all] to [none] 1> [2018-05-05T08:43:26,425][INFO ][o.e.c.s.ClusterSettings ] [node_s0] updating [cluster.routing.rebalance.enable] from [all] to [none] 1> [2018-05-05T08:43:26,431][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [target] creating index, cause [shrink_index], templates [], shards [1]/[0], mappings [] 1> [2018-05-05T08:43:26,659][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [target/7V34DuF5SMCVituHlcwyPg] create_mapping [type] 1> [2018-05-05T08:43:26,946][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[target][0]] ...]). 1> [2018-05-05T08:43:26,997][INFO ][o.e.a.a.i.c.ShrinkIndexIT] merge node {node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041} 1> [2018-05-05T08:43:27,043][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [1] for indices [target] 1> [2018-05-05T08:43:27,415][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[target][0]] ...]). 1> [2018-05-05T08:43:27,960][INFO ][o.e.c.s.ClusterSettings ] [node_sc2] updating [cluster.routing.rebalance.enable] from [none] to [all] 1> [2018-05-05T08:43:27,962][INFO ][o.e.c.s.ClusterSettings ] [node_s1] updating [cluster.routing.rebalance.enable] from [none] to [all] 1> [2018-05-05T08:43:28,065][INFO ][o.e.c.s.ClusterSettings ] [node_s0] updating [cluster.routing.rebalance.enable] from [none] to [all] 1> [2018-05-05T08:43:28,070][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testCreateShrinkIndex]: cleaning up after test 1> [2018-05-05T08:43:28,091][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [target/7V34DuF5SMCVituHlcwyPg] deleting index 1> [2018-05-05T08:43:28,091][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [source/ZYmXwyQbSPCRp1ZZtc33OA] deleting index 1> [2018-05-05T08:43:28,218][WARN ][o.e.i.IndicesService ] [node_s0] [source/ZYmXwyQbSPCRp1ZZtc33OA] failed to delete index 1> java.io.IOException: could not remove the following files (in the order of attempts): 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\3\index\_0.cfs: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\3\index\_0.cfs 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\3\index: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\3\index 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\3: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\3 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\1\index\_0.cfs: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\1\index\_0.cfs 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\1\index: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\1\index 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\1: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\1 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\2\index\_0.cfs: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\2\index\_0.cfs 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\2\index: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\2\index 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\2: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\2 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\0\index\_0.cfs: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\0\index\_0.cfs 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\0\index: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\0\index 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\0: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA\0 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\ZYmXwyQbSPCRp1ZZtc33OA 1> at org.elasticsearch.core.internal.io.IOUtils.rm(IOUtils.java:187) ~[elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectoryUnderLock(NodeEnvironment.java:505) ~[main/:?] 1> at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectorySafe(NodeEnvironment.java:489) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.deleteIndexStoreIfDeletionAllowed(IndicesService.java:668) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.deleteIndexStore(IndicesService.java:655) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:561) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:284) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:218) ~[main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:496) ~[main/:?] 1> at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_162] 1> at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:493) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:480) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:431) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:161) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:43:28,289][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [random_index_template] 1> [2018-05-05T08:43:28,311][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testCreateShrinkIndex]: cleaned up after test 1> [2018-05-05T08:43:28,312][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [testCreateShrinkIndex]: after test 1> [2018-05-05T08:43:28,345][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [testCreateShrinkIndexToN]: before test 1> [2018-05-05T08:43:28,345][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testCreateShrinkIndexToN]: setting up test 1> [2018-05-05T08:43:28,362][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [random_index_template] for index patterns [*] 1> [2018-05-05T08:43:28,375][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testCreateShrinkIndexToN]: all set up test 1> [2018-05-05T08:43:28,375][INFO ][o.e.a.a.i.c.ShrinkIndexIT] using custom data_path for index: [rzgzdlSGGQ] 1> [2018-05-05T08:43:28,377][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [source] creating index, cause [api], templates [random_index_template], shards [9]/[1], mappings [] 1> [2018-05-05T08:43:29,425][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][8]] ...]). 1> [2018-05-05T08:43:29,455][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [source/hb6ACIEAR1KRY64FNVDLTQ] create_mapping [t1] 1> [2018-05-05T08:43:29,673][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [first_shrink] creating index, cause [shrink_index], templates [], shards [3]/[0], mappings [] 1> [2018-05-05T08:43:29,977][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [first_shrink/VN4gMqYtRFWHyYJe3OfZFg] create_mapping [t1] 1> [2018-05-05T08:43:30,311][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[first_shrink][1]] ...]). 1> [2018-05-05T08:43:31,304][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [second_shrink] creating index, cause [shrink_index], templates [], shards [1]/[0], mappings [] 1> [2018-05-05T08:43:31,375][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [second_shrink/3-Xf7xL5RxGGK98H7s0LYg] create_mapping [t1] 1> [2018-05-05T08:43:31,564][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[second_shrink][0]] ...]). 1> [2018-05-05T08:43:31,907][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node_s0] updating number_of_replicas to [1] for indices [second_shrink] 1> [2018-05-05T08:43:32,574][WARN ][o.e.i.s.Store ] [node_s0] [second_shrink][0] failed to build store metadata. checking segment info integrity (with commit [no]) 1> java.nio.file.NoSuchFileException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\3-Xf7xL5RxGGK98H7s0LYg\0\index\_0.cfs 1> at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) ~[?:?] 1> at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115) ~[?:?] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.DisableFsyncFS.newFileChannel(DisableFsyncFS.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[?:1.8.0_162] 1> at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[?:1.8.0_162] 1> at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:912) ~[main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:878) [main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:797) [main/:?] 1> at org.elasticsearch.index.store.Store.getMetadata(Store.java:293) [main/:?] 1> at org.elasticsearch.index.store.Store.getMetadata(Store.java:258) [main/:?] 1> at org.elasticsearch.index.store.Store.cleanupAndVerify(Store.java:689) [main/:?] 1> at org.elasticsearch.indices.recovery.RecoveryTarget.cleanFiles(RecoveryTarget.java:440) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$CleanFilesRequestHandler.messageReceived(PeerRecoveryTargetService.java:556) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$CleanFilesRequestHandler.messageReceived(PeerRecoveryTargetService.java:550) [main/:?] 1> at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) [main/:?] 1> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [main/:?] 1> at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1603) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [main/:?] 1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:43:32,583][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[second_shrink][0]] marking and sending shard failed due to [failed recovery] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [second_shrink][0]: Recovery failed from {node_s1}{g9obCv-7RBCJawEoUGOH2Q}{DOOjJOB4Sae-PCzpbBgazA}{127.0.0.1}{127.0.0.1:60043} into {node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041} (failed to clean after recovery) 1> at org.elasticsearch.indices.recovery.RecoveryTarget.cleanFiles(RecoveryTarget.java:468) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$CleanFilesRequestHandler.messageReceived(PeerRecoveryTargetService.java:556) [main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$CleanFilesRequestHandler.messageReceived(PeerRecoveryTargetService.java:550) [main/:?] 1> at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) [main/:?] 1> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [main/:?] 1> at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1603) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [main/:?] 1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: java.nio.file.NoSuchFileException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\3-Xf7xL5RxGGK98H7s0LYg\0\index\_0.cfs 1> at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) ~[?:?] 1> at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115) ~[?:?] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.DisableFsyncFS.newFileChannel(DisableFsyncFS.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[?:1.8.0_162] 1> at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[?:1.8.0_162] 1> at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:912) ~[main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:878) ~[main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:797) ~[main/:?] 1> at org.elasticsearch.index.store.Store.getMetadata(Store.java:293) ~[main/:?] 1> at org.elasticsearch.index.store.Store.getMetadata(Store.java:258) ~[main/:?] 1> at org.elasticsearch.index.store.Store.cleanupAndVerify(Store.java:689) ~[main/:?] 1> at org.elasticsearch.indices.recovery.RecoveryTarget.cleanFiles(RecoveryTarget.java:440) [main/:?] 1> ... 10 more 1> [2018-05-05T08:43:32,586][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [second_shrink][0], node[_jxN82x-QuiKqw6LSus-lg], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=-2jKD1OWRSajOoeGTkPbTQ], unassigned_info[[reason=REPLICA_ADDED], at[2018-05-05T08:43:31.907Z], delayed=false, allocation_status[no_attempt]], message [failed recovery], failure [RecoveryFailedException[[second_shrink][0]: Recovery failed from {node_s1}{g9obCv-7RBCJawEoUGOH2Q}{DOOjJOB4Sae-PCzpbBgazA}{127.0.0.1}{127.0.0.1:60043} into {node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041} (failed to clean after recovery)]; nested: NoSuchFileException[C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\3-Xf7xL5RxGGK98H7s0LYg\0\index\_0.cfs]; ], markAsStale [true]] 1> org.elasticsearch.indices.recovery.RecoveryFailedException: [second_shrink][0]: Recovery failed from {node_s1}{g9obCv-7RBCJawEoUGOH2Q}{DOOjJOB4Sae-PCzpbBgazA}{127.0.0.1}{127.0.0.1:60043} into {node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041} (failed to clean after recovery) 1> at org.elasticsearch.indices.recovery.RecoveryTarget.cleanFiles(RecoveryTarget.java:468) ~[main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$CleanFilesRequestHandler.messageReceived(PeerRecoveryTargetService.java:556) ~[main/:?] 1> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$CleanFilesRequestHandler.messageReceived(PeerRecoveryTargetService.java:550) ~[main/:?] 1> at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) ~[main/:?] 1> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[main/:?] 1> at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1603) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: java.nio.file.NoSuchFileException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\3-Xf7xL5RxGGK98H7s0LYg\0\index\_0.cfs 1> at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[?:?] 1> at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) ~[?:?] 1> at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115) ~[?:?] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.DisableFsyncFS.newFileChannel(DisableFsyncFS.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[?:1.8.0_162] 1> at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[?:1.8.0_162] 1> at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:99) ~[lucene-core-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:13] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:912) ~[main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:878) ~[main/:?] 1> at org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:797) ~[main/:?] 1> at org.elasticsearch.index.store.Store.getMetadata(Store.java:293) ~[main/:?] 1> at org.elasticsearch.index.store.Store.getMetadata(Store.java:258) ~[main/:?] 1> at org.elasticsearch.index.store.Store.cleanupAndVerify(Store.java:689) ~[main/:?] 1> at org.elasticsearch.indices.recovery.RecoveryTarget.cleanFiles(RecoveryTarget.java:440) ~[main/:?] 1> ... 10 more 1> [2018-05-05T08:44:01,984][INFO ][o.e.a.a.i.c.ShrinkIndexIT] ensureGreen timed out, cluster state: 1> cluster uuid: abce8mRCSxeA-FXcKDHuow 1> version: 47 1> state uuid: 9qbJK-G_RB2TP_Zi2IDD6g 1> from_diff: false 1> meta data version: 42 1> [source/hb6ACIEAR1KRY64FNVDLTQ]: v[9] 1> 0: p_term [1], isa_ids [E2j5onppSRmT-ZKvf3Pu2A, FgH1BXk1Rli4xNnITkhUFQ] 1> 1: p_term [1], isa_ids [8CdW9_b-RN-WQMzGo8q8pw, aD9AFgR5RKaIp9WUBXWQ5g] 1> 2: p_term [1], isa_ids [CtmCKDI6SDeDC2a_j_Klrw, k6wgWWizQTKolDLVVsYd3w] 1> 3: p_term [1], isa_ids [tJc5ti5dSCqnkESw08QPdQ, GnGOlY1PSxuRUfSaUHFGWA] 1> 4: p_term [1], isa_ids [FDTd-jb-Q-GEOa16BN5roQ, G85GVqS8QxaXlv7WfhssCQ] 1> 5: p_term [1], isa_ids [TemlNAA8QGah5Zbx0ocqlA, KdJIDb67QZymMgk-_9W5Zg] 1> 6: p_term [1], isa_ids [0F7O0m4eRRqVjfyEXs6eJw, qsWTn3ESQlaYbfqvyrPnCw] 1> 7: p_term [1], isa_ids [bRC4tcdKRb24luDbytFg3w, V8zSM1S-TJi2ivf0rz7F8w] 1> 8: p_term [1], isa_ids [obYICnBHSYWQKxrADECRNw, Rm0CRPTaSa-0Av-EQXa-6w] 1> [second_shrink/3-Xf7xL5RxGGK98H7s0LYg]: v[7] 1> 0: p_term [3], isa_ids [DKFqCm_RQxue8_6g3GL4Tw] 1> [first_shrink/VN4gMqYtRFWHyYJe3OfZFg]: v[9] 1> 0: p_term [2], isa_ids [Jl5cr5J2TPyF3fYt9_1pjQ] 1> 1: p_term [2], isa_ids [IN1O30JsRWyUwPJsRZuXxA] 1> 2: p_term [2], isa_ids [WGEMSW_URouTPywpCHVTkQ] 1> blocks: 1> first_shrink: 1> 8,index write (api), blocks WRITE source: 1> 8,index write (api), blocks WRITE 1> nodes: 1> {node_sc2}{9vx9IZB3Tz2BJpMfrMcqqA}{nb6v1D7YSUmzgPbW4eRV4w}{127.0.0.1}{127.0.0.1:60045} 1> {node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041}, master 1> {node_s1}{g9obCv-7RBCJawEoUGOH2Q}{DOOjJOB4Sae-PCzpbBgazA}{127.0.0.1}{127.0.0.1:60043} 1> routing_table (version 34): 1> -- index [[source/hb6ACIEAR1KRY64FNVDLTQ]] 1> ----shard_id [source][0] 1> --------[source][0], node[g9obCv-7RBCJawEoUGOH2Q], [P], s[STARTED], a[id=FgH1BXk1Rli4xNnITkhUFQ] 1> --------[source][0], node[_jxN82x-QuiKqw6LSus-lg], [R], s[STARTED], a[id=E2j5onppSRmT-ZKvf3Pu2A] 1> ----shard_id [source][1] 1> --------[source][1], node[g9obCv-7RBCJawEoUGOH2Q], [R], s[STARTED], a[id=aD9AFgR5RKaIp9WUBXWQ5g] 1> --------[source][1], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=8CdW9_b-RN-WQMzGo8q8pw] 1> ----shard_id [source][2] 1> --------[source][2], node[g9obCv-7RBCJawEoUGOH2Q], [P], s[STARTED], a[id=CtmCKDI6SDeDC2a_j_Klrw] 1> --------[source][2], node[_jxN82x-QuiKqw6LSus-lg], [R], s[STARTED], a[id=k6wgWWizQTKolDLVVsYd3w] 1> ----shard_id [source][3] 1> --------[source][3], node[g9obCv-7RBCJawEoUGOH2Q], [R], s[STARTED], a[id=GnGOlY1PSxuRUfSaUHFGWA] 1> --------[source][3], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=tJc5ti5dSCqnkESw08QPdQ] 1> ----shard_id [source][4] 1> --------[source][4], node[g9obCv-7RBCJawEoUGOH2Q], [P], s[STARTED], a[id=FDTd-jb-Q-GEOa16BN5roQ] 1> --------[source][4], node[_jxN82x-QuiKqw6LSus-lg], [R], s[STARTED], a[id=G85GVqS8QxaXlv7WfhssCQ] 1> ----shard_id [source][5] 1> --------[source][5], node[g9obCv-7RBCJawEoUGOH2Q], [R], s[STARTED], a[id=TemlNAA8QGah5Zbx0ocqlA] 1> --------[source][5], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=KdJIDb67QZymMgk-_9W5Zg] 1> ----shard_id [source][6] 1> --------[source][6], node[g9obCv-7RBCJawEoUGOH2Q], [P], s[STARTED], a[id=qsWTn3ESQlaYbfqvyrPnCw] 1> --------[source][6], node[_jxN82x-QuiKqw6LSus-lg], [R], s[STARTED], a[id=0F7O0m4eRRqVjfyEXs6eJw] 1> ----shard_id [source][7] 1> --------[source][7], node[g9obCv-7RBCJawEoUGOH2Q], [R], s[STARTED], a[id=bRC4tcdKRb24luDbytFg3w] 1> --------[source][7], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=V8zSM1S-TJi2ivf0rz7F8w] 1> ----shard_id [source][8] 1> --------[source][8], node[g9obCv-7RBCJawEoUGOH2Q], [P], s[STARTED], a[id=Rm0CRPTaSa-0Av-EQXa-6w] 1> --------[source][8], node[_jxN82x-QuiKqw6LSus-lg], [R], s[STARTED], a[id=obYICnBHSYWQKxrADECRNw] 1> -- index [[second_shrink/3-Xf7xL5RxGGK98H7s0LYg]] 1> ----shard_id [second_shrink][0] 1> --------[second_shrink][0], node[g9obCv-7RBCJawEoUGOH2Q], [P], s[STARTED], a[id=DKFqCm_RQxue8_6g3GL4Tw] 1> --------[second_shrink][0], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:43:32.586Z], failed_attempts[1], delayed=false, details[failed shard on node [_jxN82x-QuiKqw6LSus-lg]: failed recovery, failure RecoveryFailedException[[second_shrink][0]: Recovery failed from {node_s1}{g9obCv-7RBCJawEoUGOH2Q}{DOOjJOB4Sae-PCzpbBgazA}{127.0.0.1}{127.0.0.1:60043} into {node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041} (failed to clean after recovery)]; nested: NoSuchFileException[C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\3-Xf7xL5RxGGK98H7s0LYg\0\index\_0.cfs]; ], allocation_status[no_attempt]] 1> -- index [[first_shrink/VN4gMqYtRFWHyYJe3OfZFg]] 1> ----shard_id [first_shrink][0] 1> --------[first_shrink][0], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=Jl5cr5J2TPyF3fYt9_1pjQ] 1> ----shard_id [first_shrink][1] 1> --------[first_shrink][1], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=IN1O30JsRWyUwPJsRZuXxA] 1> ----shard_id [first_shrink][2] 1> --------[first_shrink][2], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=WGEMSW_URouTPywpCHVTkQ] 1> routing_nodes: 1> -----node_id[g9obCv-7RBCJawEoUGOH2Q][V] 1> --------[source][6], node[g9obCv-7RBCJawEoUGOH2Q], [P], s[STARTED], a[id=qsWTn3ESQlaYbfqvyrPnCw] 1> --------[source][1], node[g9obCv-7RBCJawEoUGOH2Q], [R], s[STARTED], a[id=aD9AFgR5RKaIp9WUBXWQ5g] 1> --------[source][7], node[g9obCv-7RBCJawEoUGOH2Q], [R], s[STARTED], a[id=bRC4tcdKRb24luDbytFg3w] 1> --------[source][4], node[g9obCv-7RBCJawEoUGOH2Q], [P], s[STARTED], a[id=FDTd-jb-Q-GEOa16BN5roQ] 1> --------[source][3], node[g9obCv-7RBCJawEoUGOH2Q], [R], s[STARTED], a[id=GnGOlY1PSxuRUfSaUHFGWA] 1> --------[source][5], node[g9obCv-7RBCJawEoUGOH2Q], [R], s[STARTED], a[id=TemlNAA8QGah5Zbx0ocqlA] 1> --------[source][8], node[g9obCv-7RBCJawEoUGOH2Q], [P], s[STARTED], a[id=Rm0CRPTaSa-0Av-EQXa-6w] 1> --------[source][2], node[g9obCv-7RBCJawEoUGOH2Q], [P], s[STARTED], a[id=CtmCKDI6SDeDC2a_j_Klrw] 1> --------[source][0], node[g9obCv-7RBCJawEoUGOH2Q], [P], s[STARTED], a[id=FgH1BXk1Rli4xNnITkhUFQ] 1> --------[second_shrink][0], node[g9obCv-7RBCJawEoUGOH2Q], [P], s[STARTED], a[id=DKFqCm_RQxue8_6g3GL4Tw] 1> -----node_id[_jxN82x-QuiKqw6LSus-lg][V] 1> --------[source][6], node[_jxN82x-QuiKqw6LSus-lg], [R], s[STARTED], a[id=0F7O0m4eRRqVjfyEXs6eJw] 1> --------[source][1], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=8CdW9_b-RN-WQMzGo8q8pw] 1> --------[source][7], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=V8zSM1S-TJi2ivf0rz7F8w] 1> --------[source][4], node[_jxN82x-QuiKqw6LSus-lg], [R], s[STARTED], a[id=G85GVqS8QxaXlv7WfhssCQ] 1> --------[source][3], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=tJc5ti5dSCqnkESw08QPdQ] 1> --------[source][5], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=KdJIDb67QZymMgk-_9W5Zg] 1> --------[source][8], node[_jxN82x-QuiKqw6LSus-lg], [R], s[STARTED], a[id=obYICnBHSYWQKxrADECRNw] 1> --------[source][2], node[_jxN82x-QuiKqw6LSus-lg], [R], s[STARTED], a[id=k6wgWWizQTKolDLVVsYd3w] 1> --------[source][0], node[_jxN82x-QuiKqw6LSus-lg], [R], s[STARTED], a[id=E2j5onppSRmT-ZKvf3Pu2A] 1> --------[first_shrink][2], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=WGEMSW_URouTPywpCHVTkQ] 1> --------[first_shrink][1], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=IN1O30JsRWyUwPJsRZuXxA] 1> --------[first_shrink][0], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=Jl5cr5J2TPyF3fYt9_1pjQ] 1> ---- unassigned 1> --------[second_shrink][0], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-05-05T08:43:32.586Z], failed_attempts[1], delayed=false, details[failed shard on node [_jxN82x-QuiKqw6LSus-lg]: failed recovery, failure RecoveryFailedException[[second_shrink][0]: Recovery failed from {node_s1}{g9obCv-7RBCJawEoUGOH2Q}{DOOjJOB4Sae-PCzpbBgazA}{127.0.0.1}{127.0.0.1:60043} into {node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041} (failed to clean after recovery)]; nested: NoSuchFileException[C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\0\indices\3-Xf7xL5RxGGK98H7s0LYg\0\index\_0.cfs]; ], allocation_status[no_attempt]] 1> customs: 1> restore: RestoreInProgress[] snapshots: SnapshotsInProgress[] snapshot_deletions: SnapshotDeletionsInProgress[] 1> tasks: (0): 1> 1> [2018-05-05T08:44:01,985][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testCreateShrinkIndexToN]: cleaning up after test 1> [2018-05-05T08:44:01,999][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [second_shrink/3-Xf7xL5RxGGK98H7s0LYg] deleting index 1> [2018-05-05T08:44:01,999][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [first_shrink/VN4gMqYtRFWHyYJe3OfZFg] deleting index 1> [2018-05-05T08:44:01,999][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [source/hb6ACIEAR1KRY64FNVDLTQ] deleting index 1> [2018-05-05T08:44:02,210][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [random_index_template] 1> [2018-05-05T08:44:02,215][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testCreateShrinkIndexToN]: cleaned up after test 1> [2018-05-05T08:44:02,215][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [testCreateShrinkIndexToN]: after test 2> REPRODUCE WITH: gradlew :server:integTest -Dtests.seed=734B52BB31887C24 -Dtests.class=org.elasticsearch.action.admin.indices.create.ShrinkIndexIT -Dtests.method="testCreateShrinkIndexToN" -Dtests.security.manager=true -Dtests.locale=sk-SK -Dtests.timezone=UCT FAILURE 33.9s J0 | ShrinkIndexIT.testCreateShrinkIndexToN <<< FAILURES! > Throwable #1: java.lang.AssertionError: timed out waiting for green state > at __randomizedtesting.SeedInfo.seed([734B52BB31887C24:630B1C77942837B7]:0) > at org.elasticsearch.test.ESIntegTestCase.ensureColor(ESIntegTestCase.java:956) > at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:912) > at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:901) > at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testCreateShrinkIndexToN(ShrinkIndexIT.java:146) > at java.lang.Thread.run(Thread.java:748) 1> [2018-05-05T08:44:02,253][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [testShrinkCommitsMergeOnIdle]: before test 1> [2018-05-05T08:44:02,253][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testShrinkCommitsMergeOnIdle]: setting up test 1> [2018-05-05T08:44:02,258][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [random_index_template] for index patterns [*] 1> [2018-05-05T08:44:02,265][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testShrinkCommitsMergeOnIdle]: all set up test 1> [2018-05-05T08:44:02,265][INFO ][o.e.a.a.i.c.ShrinkIndexIT] using custom data_path for index: [rtTCVihVna] 1> [2018-05-05T08:44:02,266][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [source] creating index, cause [api], templates [random_index_template], shards [5]/[0], mappings [] 1> [2018-05-05T08:44:02,348][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][1], [source][3]] ...]). 1> [2018-05-05T08:44:02,358][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [source/aGauKb23S6qtyrzbeHANAg] create_mapping [type] 1> [2018-05-05T08:44:02,599][INFO ][o.e.c.s.ClusterSettings ] [node_sc2] updating [cluster.routing.rebalance.enable] from [all] to [none] 1> [2018-05-05T08:44:02,615][INFO ][o.e.c.s.ClusterSettings ] [node_s1] updating [cluster.routing.rebalance.enable] from [all] to [none] 1> [2018-05-05T08:44:02,616][INFO ][o.e.c.s.ClusterSettings ] [node_s0] updating [cluster.routing.rebalance.enable] from [all] to [none] 1> [2018-05-05T08:44:02,618][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [target] creating index, cause [shrink_index], templates [], shards [1]/[0], mappings [] 1> [2018-05-05T08:44:02,633][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [target/naYUxzY5SOupK-ofcqNRzw] create_mapping [type] 1> [2018-05-05T08:44:02,729][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[target][0]] ...]). 1> [2018-05-05T08:44:03,795][INFO ][o.e.c.s.ClusterSettings ] [node_sc2] updating [cluster.routing.rebalance.enable] from [none] to [all] 1> [2018-05-05T08:44:03,795][INFO ][o.e.c.s.ClusterSettings ] [node_s1] updating [cluster.routing.rebalance.enable] from [none] to [all] 1> [2018-05-05T08:44:03,797][INFO ][o.e.c.s.ClusterSettings ] [node_s0] updating [cluster.routing.rebalance.enable] from [none] to [all] 1> [2018-05-05T08:44:03,811][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testShrinkCommitsMergeOnIdle]: cleaning up after test 1> [2018-05-05T08:44:03,817][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [source/aGauKb23S6qtyrzbeHANAg] deleting index 1> [2018-05-05T08:44:03,817][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [target/naYUxzY5SOupK-ofcqNRzw] deleting index 1> [2018-05-05T08:44:03,899][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [random_index_template] 1> [2018-05-05T08:44:03,904][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testShrinkCommitsMergeOnIdle]: cleaned up after test 1> [2018-05-05T08:44:03,904][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [testShrinkCommitsMergeOnIdle]: after test 1> [2018-05-05T08:44:03,922][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [testShrinkIndexPrimaryTerm]: before test 1> [2018-05-05T08:44:03,922][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testShrinkIndexPrimaryTerm]: setting up test 1> [2018-05-05T08:44:03,927][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [random_index_template] for index patterns [*] 1> [2018-05-05T08:44:03,932][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testShrinkIndexPrimaryTerm]: all set up test 1> [2018-05-05T08:44:03,936][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [source] creating index, cause [api], templates [random_index_template], shards [5]/[1], mappings [] 1> [2018-05-05T08:44:04,108][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][3]] ...]). 1> [2018-05-05T08:44:04,123][WARN ][o.e.i.e.Engine ] [node_s1] [source][0] failed engine [test] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) [test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,125][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[source][0]] marking and sending shard failed due to [shard failure, reason [test]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,128][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [source/4s95-EKgRWK0mUYN8ePelg] create_mapping [type] 1> [2018-05-05T08:44:04,137][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [source][0], node[g9obCv-7RBCJawEoUGOH2Q], [P], s[STARTED], a[id=9FQr7WRfTh-NTpblwQdYZA], message [shard failure, reason [test]], failure [NotSerializableExceptionWrapper[exception: test]], markAsStale [true]] 1> org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,138][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [YELLOW] (reason: [shards failed [[source][0], [source][0]] ...]). 1> [2018-05-05T08:44:04,152][INFO ][o.e.i.s.IndexShard ] [node_s0] [source][0] primary-replica resync completed with 0 operations 1> [2018-05-05T08:44:04,179][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][0]] ...]). 1> [2018-05-05T08:44:04,196][WARN ][o.e.i.e.Engine ] [node_s1] [source][2] failed engine [test] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) [test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,197][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[source][2]] marking and sending shard failed due to [shard failure, reason [test]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,199][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [source][2], node[g9obCv-7RBCJawEoUGOH2Q], [P], s[STARTED], a[id=TQjbgkarTZyt3RCSUXOfjA], message [shard failure, reason [test]], failure [NotSerializableExceptionWrapper[exception: test]], markAsStale [true]] 1> org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,201][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [YELLOW] (reason: [shards failed [[source][2]] ...]). 1> [2018-05-05T08:44:04,213][INFO ][o.e.i.s.IndexShard ] [node_s0] [source][2] primary-replica resync completed with 0 operations 1> [2018-05-05T08:44:04,242][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][2]] ...]). 1> [2018-05-05T08:44:04,257][WARN ][o.e.i.e.Engine ] [node_s1] [source][4] failed engine [test] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) [test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,258][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[source][4]] marking and sending shard failed due to [shard failure, reason [test]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,261][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [source][4], node[g9obCv-7RBCJawEoUGOH2Q], [P], s[STARTED], a[id=nTr-fqCORhiY7R2_6puwgg], message [shard failure, reason [test]], failure [NotSerializableExceptionWrapper[exception: test]], markAsStale [true]] 1> org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,263][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [YELLOW] (reason: [shards failed [[source][4]] ...]). 1> [2018-05-05T08:44:04,278][INFO ][o.e.i.s.IndexShard ] [node_s0] [source][4] primary-replica resync completed with 0 operations 1> [2018-05-05T08:44:04,306][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][4]] ...]). 1> [2018-05-05T08:44:04,324][WARN ][o.e.i.e.Engine ] [node_s0] [source][0] failed engine [test] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) [test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,326][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[source][0]] marking and sending shard failed due to [shard failure, reason [test]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,328][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [source][0], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=IqVobjB-RpG7VrutTL4Igw], message [shard failure, reason [test]], failure [Exception[test]], markAsStale [true]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,330][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [YELLOW] (reason: [shards failed [[source][0]] ...]). 1> [2018-05-05T08:44:04,341][INFO ][o.e.i.s.IndexShard ] [node_s1] [source][0] primary-replica resync completed with 0 operations 1> [2018-05-05T08:44:04,413][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][0]] ...]). 1> [2018-05-05T08:44:04,438][WARN ][o.e.i.e.Engine ] [node_s0] [source][1] failed engine [test] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) [test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,439][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[source][1]] marking and sending shard failed due to [shard failure, reason [test]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,442][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [source][1], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=6-mdpIPLQkGYamoWTeRT4Q], message [shard failure, reason [test]], failure [Exception[test]], markAsStale [true]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,444][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [YELLOW] (reason: [shards failed [[source][1]] ...]). 1> [2018-05-05T08:44:04,455][INFO ][o.e.i.s.IndexShard ] [node_s1] [source][1] primary-replica resync completed with 0 operations 1> [2018-05-05T08:44:04,507][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][1]] ...]). 1> [2018-05-05T08:44:04,529][WARN ][o.e.i.e.Engine ] [node_s0] [source][2] failed engine [test] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) [test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,531][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[source][2]] marking and sending shard failed due to [shard failure, reason [test]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,534][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [source][2], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=mfOm6cC0RN-lhqVcv3za0w], message [shard failure, reason [test]], failure [Exception[test]], markAsStale [true]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,536][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [YELLOW] (reason: [shards failed [[source][2]] ...]). 1> [2018-05-05T08:44:04,543][INFO ][o.e.i.s.IndexShard ] [node_s1] [source][2] primary-replica resync completed with 0 operations 1> [2018-05-05T08:44:04,608][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][2]] ...]). 1> [2018-05-05T08:44:04,631][WARN ][o.e.i.e.Engine ] [node_s0] [source][3] failed engine [test] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) [test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,633][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[source][3]] marking and sending shard failed due to [shard failure, reason [test]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,635][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [source][3], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=qsIk_3xDRFqYy6-PHSxrpw], message [shard failure, reason [test]], failure [Exception[test]], markAsStale [true]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,637][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [YELLOW] (reason: [shards failed [[source][3]] ...]). 1> [2018-05-05T08:44:04,647][INFO ][o.e.i.s.IndexShard ] [node_s1] [source][3] primary-replica resync completed with 0 operations 1> [2018-05-05T08:44:04,721][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][3]] ...]). 1> [2018-05-05T08:44:04,776][WARN ][o.e.i.e.Engine ] [node_s0] [source][4] failed engine [test] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) [test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,778][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[source][4]] marking and sending shard failed due to [shard failure, reason [test]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,782][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [source][4], node[_jxN82x-QuiKqw6LSus-lg], [P], s[STARTED], a[id=IHe7lQ1WREOP3fBN_DfPqw], message [shard failure, reason [test]], failure [Exception[test]], markAsStale [true]] 1> java.lang.Exception: test 1> at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:186) ~[test/:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] 1> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] 1> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] 1> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-7.4.0-snapshot-1ed95c097b.jar:7.4.0-snapshot-1ed95c097b 1ed95c097b82ee5f175e93f3fe62572abe064da6 - jimczi - 2018-05-03 16:16:16] 1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.5.2.jar:?] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:04,785][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [GREEN] to [YELLOW] (reason: [shards failed [[source][4]] ...]). 1> [2018-05-05T08:44:04,799][INFO ][o.e.i.s.IndexShard ] [node_s1] [source][4] primary-replica resync completed with 0 operations 1> [2018-05-05T08:44:04,916][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][4]] ...]). 1> [2018-05-05T08:44:04,961][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [target] creating index, cause [shrink_index], templates [], shards [1]/[0], mappings [] 1> [2018-05-05T08:44:05,101][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [target/gFVoXkCLTluomdlbg1nMCw] create_mapping [type] 1> [2018-05-05T08:44:05,224][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[target][0]] ...]). 1> [2018-05-05T08:44:05,235][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testShrinkIndexPrimaryTerm]: cleaning up after test 1> [2018-05-05T08:44:05,245][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [target/gFVoXkCLTluomdlbg1nMCw] deleting index 1> [2018-05-05T08:44:05,245][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [source/4s95-EKgRWK0mUYN8ePelg] deleting index 1> [2018-05-05T08:44:05,307][WARN ][o.e.i.IndicesService ] [node_s1] [source/4s95-EKgRWK0mUYN8ePelg] failed to delete index 1> java.io.IOException: could not remove the following files (in the order of attempts): 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\3\index\_0.cfs: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\3\index\_0.cfs 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\3\index: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\3\index 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\3: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\3 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\4\index\_0.cfs: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\4\index\_0.cfs 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\4\index: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\4\index 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\4: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\4 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\0\index\_0.cfs: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\0\index\_0.cfs 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\0\index: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\0\index 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\0: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg\0 1> C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001\tempDir-002\data\nodes\1\indices\4s95-EKgRWK0mUYN8ePelg 1> at org.elasticsearch.core.internal.io.IOUtils.rm(IOUtils.java:187) ~[elasticsearch-core-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectoryUnderLock(NodeEnvironment.java:505) ~[main/:?] 1> at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectorySafe(NodeEnvironment.java:489) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.deleteIndexStoreIfDeletionAllowed(IndicesService.java:668) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.deleteIndexStore(IndicesService.java:655) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:561) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:284) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:218) ~[main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:496) ~[main/:?] 1> at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_162] 1> at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:493) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:480) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:431) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:161) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:05,322][WARN ][o.e.i.IndicesService ] [node_s1] [source/4s95-EKgRWK0mUYN8ePelg] still pending deletes present for shards [[[source/4s95-EKgRWK0mUYN8ePelg]], [[source/4s95-EKgRWK0mUYN8ePelg]][0], [[source/4s95-EKgRWK0mUYN8ePelg]][3]] - retrying 1> [2018-05-05T08:44:05,431][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [random_index_template] 1> [2018-05-05T08:44:05,439][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testShrinkIndexPrimaryTerm]: cleaned up after test 1> [2018-05-05T08:44:05,439][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [testShrinkIndexPrimaryTerm]: after test 1> [2018-05-05T08:44:05,461][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [testCreateShrinkWithIndexSort]: before test 1> [2018-05-05T08:44:05,461][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testCreateShrinkWithIndexSort]: setting up test 1> [2018-05-05T08:44:05,466][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [random_index_template] for index patterns [*] 1> [2018-05-05T08:44:05,474][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testCreateShrinkWithIndexSort]: all set up test 1> [2018-05-05T08:44:05,477][INFO ][o.e.a.a.i.c.ShrinkIndexIT] using custom data_path for index: [PQbbSYMBnv] 1> [2018-05-05T08:44:05,489][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [source] creating index, cause [api], templates [random_index_template], shards [8]/[0], mappings [type] 1> [2018-05-05T08:44:05,628][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][1], [source][5]] ...]). 1> [2018-05-05T08:44:05,639][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [source/X9V5jd1lQDSFF_sGgSwHWg] update_mapping [type] 1> [2018-05-05T08:44:06,095][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [target] creating index, cause [shrink_index], templates [], shards [2]/[0], mappings [] 1> [2018-05-05T08:44:06,129][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [target/IyGz3bz8Q1CHQJr-29wGpg] create_mapping [type] 1> [2018-05-05T08:44:06,309][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[target][1]] ...]). 1> [2018-05-05T08:44:06,528][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [target/IyGz3bz8Q1CHQJr-29wGpg] update_mapping [type] 1> [2018-05-05T08:44:06,608][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testCreateShrinkWithIndexSort]: cleaning up after test 1> [2018-05-05T08:44:06,621][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [source/X9V5jd1lQDSFF_sGgSwHWg] deleting index 1> [2018-05-05T08:44:06,621][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [target/IyGz3bz8Q1CHQJr-29wGpg] deleting index 1> [2018-05-05T08:44:06,806][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [random_index_template] 1> [2018-05-05T08:44:06,814][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testCreateShrinkWithIndexSort]: cleaned up after test 1> [2018-05-05T08:44:06,814][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [testCreateShrinkWithIndexSort]: after test 1> [2018-05-05T08:44:06,837][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [testCreateShrinkIndexFails]: before test 1> [2018-05-05T08:44:06,837][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testCreateShrinkIndexFails]: setting up test 1> [2018-05-05T08:44:06,844][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [random_index_template] for index patterns [*] 1> [2018-05-05T08:44:06,851][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testCreateShrinkIndexFails]: all set up test 1> [2018-05-05T08:44:06,852][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [source] creating index, cause [api], templates [random_index_template], shards [5]/[0], mappings [] 1> [2018-05-05T08:44:06,937][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[source][3], [source][0]] ...]). 1> [2018-05-05T08:44:06,952][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [source/4JmNSswuSOKMGf260vBjBA] create_mapping [type] 1> [2018-05-05T08:44:07,210][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [target] creating index, cause [shrink_index], templates [], shards [1]/[0], mappings [] 1> [2018-05-05T08:44:07,211][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [RED] (reason: [index [target] created]). 1> [2018-05-05T08:44:07,527][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][0]] marking and sending shard failed due to [failed to create shard] 1> org.elasticsearch.index.IndexNotFoundException: no such index 1> at org.elasticsearch.index.shard.IndexShard.startRecovery(IndexShard.java:2054) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:517) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:144) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.createShard(IndicesClusterStateService.java:550) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:527) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:230) ~[main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:496) ~[main/:?] 1> at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_162] 1> at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:493) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:480) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:431) [main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:161) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:07,539][WARN ][o.e.c.r.a.AllocationService] [node_s0] failing shard [failed shard, shard [target][0], node[g9obCv-7RBCJawEoUGOH2Q], [P], recovery_source[local shards recovery], s[INITIALIZING], a[id=tcWfZ5X1SfK5dgwvbKJobg], unassigned_info[[reason=INDEX_CREATED], at[2018-05-05T08:44:07.210Z], delayed=false, allocation_status[deciders_no]], expected_shard_size[13130], message [failed to create shard], failure [IndexNotFoundException[no such index]], markAsStale [true]] 1> org.elasticsearch.index.IndexNotFoundException: no such index 1> at org.elasticsearch.index.shard.IndexShard.startRecovery(IndexShard.java:2054) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:517) ~[main/:?] 1> at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:144) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.createShard(IndicesClusterStateService.java:550) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:527) ~[main/:?] 1> at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:230) ~[main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:496) ~[main/:?] 1> at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_162] 1> at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:493) ~[main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:480) ~[main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:431) ~[main/:?] 1> at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:161) ~[main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [main/:?] 1> at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> [2018-05-05T08:44:07,813][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [target/o_76D-hbT1y8gL8oNfRmwA] create_mapping [type] 1> [2018-05-05T08:44:07,921][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[target][0]] ...]). 1> [2018-05-05T08:44:08,045][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testCreateShrinkIndexFails]: cleaning up after test 1> [2018-05-05T08:44:08,055][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [source/4JmNSswuSOKMGf260vBjBA] deleting index 1> [2018-05-05T08:44:08,055][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [target/o_76D-hbT1y8gL8oNfRmwA] deleting index 1> [2018-05-05T08:44:08,118][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [random_index_template] 1> [2018-05-05T08:44:08,125][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [ShrinkIndexIT#testCreateShrinkIndexFails]: cleaned up after test 1> [2018-05-05T08:44:08,125][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [testCreateShrinkIndexFails]: after test 1> [2018-05-05T08:44:08,127][INFO ][o.e.n.Node ] [node_s0] stopping ... 1> [2018-05-05T08:44:08,128][INFO ][o.e.t.d.TestZenDiscovery ] [node_s1] master_left [{node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041}], reason [shut_down] 1> [2018-05-05T08:44:08,128][WARN ][o.e.t.d.TestZenDiscovery ] [node_s1] master left (reason = shut_down), current nodes: nodes: 1> {node_sc2}{9vx9IZB3Tz2BJpMfrMcqqA}{nb6v1D7YSUmzgPbW4eRV4w}{127.0.0.1}{127.0.0.1:60045} 1> {node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041}, master 1> {node_s1}{g9obCv-7RBCJawEoUGOH2Q}{DOOjJOB4Sae-PCzpbBgazA}{127.0.0.1}{127.0.0.1:60043}, local 1> 1> [2018-05-05T08:44:08,128][INFO ][o.e.c.s.ClusterApplierService] [node_s1] master node changed {previous [{node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041}], current []}, reason: master left (reason = shut_down) 1> [2018-05-05T08:44:08,129][INFO ][o.e.t.d.TestZenDiscovery ] [node_sc2] master_left [{node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041}], reason [transport disconnected] 1> [2018-05-05T08:44:08,130][WARN ][o.e.t.d.TestZenDiscovery ] [node_sc2] master left (reason = transport disconnected), current nodes: nodes: 1> {node_sc2}{9vx9IZB3Tz2BJpMfrMcqqA}{nb6v1D7YSUmzgPbW4eRV4w}{127.0.0.1}{127.0.0.1:60045}, local 1> {node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041}, master 1> {node_s1}{g9obCv-7RBCJawEoUGOH2Q}{DOOjJOB4Sae-PCzpbBgazA}{127.0.0.1}{127.0.0.1:60043} 1> 1> [2018-05-05T08:44:08,130][INFO ][o.e.c.s.ClusterApplierService] [node_sc2] master node changed {previous [{node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041}], current []}, reason: master left (reason = transport disconnected) 1> [2018-05-05T08:44:08,129][INFO ][o.e.t.d.MockZenPing ] [node_s1] pinging using mock zen ping 1> [2018-05-05T08:44:08,130][INFO ][o.e.t.d.MockZenPing ] [node_sc2] pinging using mock zen ping 1> [2018-05-05T08:44:08,133][INFO ][o.e.n.Node ] [node_s0] stopped 1> [2018-05-05T08:44:08,133][INFO ][o.e.n.Node ] [node_s0] closing ... 1> [2018-05-05T08:44:08,134][INFO ][o.e.n.Node ] [node_s0] closed 1> [2018-05-05T08:44:08,136][INFO ][o.e.n.Node ] [node_s1] stopping ... 1> [2018-05-05T08:44:08,136][WARN ][o.e.t.d.TestZenDiscovery ] [node_sc2] not enough master nodes discovered during pinging (found [[]], but needed [2]), pinging again 1> [2018-05-05T08:44:08,136][WARN ][o.e.t.d.TestZenDiscovery ] [node_s1] not enough master nodes discovered during pinging (found [[Candidate{node={node_s1}{g9obCv-7RBCJawEoUGOH2Q}{DOOjJOB4Sae-PCzpbBgazA}{127.0.0.1}{127.0.0.1:60043}, clusterStateVersion=169}]], but needed [2]), pinging again 1> [2018-05-05T08:44:08,136][INFO ][o.e.t.d.MockZenPing ] [node_sc2] pinging using mock zen ping 1> [2018-05-05T08:44:09,131][WARN ][o.e.c.NodeConnectionsService] [node_s1] failed to connect to node {node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041} (tried [1] times) 1> org.elasticsearch.transport.ConnectTransportException: [node_s0][127.0.0.1:60041] connect_exception 1> at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:177) ~[main/:?] 1> at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:615) ~[main/:?] 1> at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:514) ~[main/:?] 1> at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:331) ~[main/:?] 1> at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:318) ~[main/:?] 1> at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) [main/:?] 1> at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [main/:?] 1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: java.net.ConnectException: Connection refused: no further information 1> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] 1> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?] 1> at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:98) ~[elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.nio.SocketEventHandler.handleConnect(SocketEventHandler.java:76) ~[elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.transport.nio.TestingSocketEventHandler.handleConnect(TestingSocketEventHandler.java:41) ~[framework-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.nio.SocketSelector.attemptConnect(SocketSelector.java:216) ~[elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.nio.SocketSelector.processKey(SocketSelector.java:55) ~[elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.nio.ESSelector.singleLoop(ESSelector.java:112) ~[elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.nio.ESSelector.runLoop(ESSelector.java:77) ~[elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> ... 1 more 1> [2018-05-05T08:44:09,132][WARN ][o.e.c.NodeConnectionsService] [node_sc2] failed to connect to node {node_s0}{_jxN82x-QuiKqw6LSus-lg}{FIouDb2wQDyf0TVyEtuJZw}{127.0.0.1}{127.0.0.1:60041} (tried [1] times) 1> org.elasticsearch.transport.ConnectTransportException: [node_s0][127.0.0.1:60041] connect_exception 1> at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:177) ~[main/:?] 1> at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:615) ~[main/:?] 1> at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:514) ~[main/:?] 1> at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:331) ~[main/:?] 1> at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:318) ~[main/:?] 1> at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) [main/:?] 1> at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) [main/:?] 1> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [main/:?] 1> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [main/:?] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] 1> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] 1> Caused by: java.net.ConnectException: Connection refused: no further information 1> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] 1> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?] 1> at org.elasticsearch.nio.SocketChannelContext.connect(SocketChannelContext.java:98) ~[elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.nio.SocketEventHandler.handleConnect(SocketEventHandler.java:76) ~[elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.transport.nio.TestingSocketEventHandler.handleConnect(TestingSocketEventHandler.java:41) ~[framework-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.nio.SocketSelector.attemptConnect(SocketSelector.java:216) ~[elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.nio.SocketSelector.processKey(SocketSelector.java:55) ~[elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.nio.ESSelector.singleLoop(ESSelector.java:112) ~[elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> at org.elasticsearch.nio.ESSelector.runLoop(ESSelector.java:77) ~[elasticsearch-nio-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 1> ... 1 more 1> [2018-05-05T08:44:09,137][INFO ][o.e.n.Node ] [node_s1] stopped 1> [2018-05-05T08:44:09,137][INFO ][o.e.n.Node ] [node_s1] closing ... 1> [2018-05-05T08:44:09,139][INFO ][o.e.n.Node ] [node_s1] closed 1> [2018-05-05T08:44:09,141][INFO ][o.e.n.Node ] [node_sc2] stopping ... 1> [2018-05-05T08:44:09,141][WARN ][o.e.t.d.TestZenDiscovery ] [node_sc2] not enough master nodes discovered during pinging (found [[]], but needed [2]), pinging again 1> [2018-05-05T08:44:09,142][INFO ][o.e.n.Node ] [node_sc2] stopped 1> [2018-05-05T08:44:09,142][INFO ][o.e.n.Node ] [node_sc2] closing ... 1> [2018-05-05T08:44:09,144][INFO ][o.e.n.Node ] [node_sc2] closed 2> NOTE: leaving temporary files on disk at: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\server\build\testrun\integTest\J0\temp\org.elasticsearch.action.admin.indices.create.ShrinkIndexIT_734B52BB31887C24-001 2> NOTE: test params are: codec=Asserting(Lucene70): {f=PostingsFormat(name=Asserting), foo=PostingsFormat(name=Asserting), _id=PostingsFormat(name=Asserting), id=PostingsFormat(name=Asserting), f.keyword=PostingsFormat(name=Asserting), foo.keyword=PostingsFormat(name=Asserting)}, docValues:{_seq_no=DocValuesFormat(name=Asserting), i=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Asserting), _primary_term=DocValuesFormat(name=Lucene70), _version=DocValuesFormat(name=Lucene70), f.keyword=DocValuesFormat(name=Lucene70), foo.keyword=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1529, maxMBSortInHeap=6.584448579361242, sim=RandomSimilarity(queryNorm=false): {}, locale=sk-SK, timezone=UCT 2> NOTE: Windows Server 2016 10.0 amd64/Oracle Corporation 1.8.0_162 (64-bit)/cpus=4,threads=1,free=292875344,total=518520832 2> NOTE: All tests run in this JVM: [HighlighterSearchIT, SuggestSearchIT, GeoBoundingBoxIT, InternalEngineMergeIT, CorruptedFileIT, SimpleBlocksIT, ClusterSearchShardsIT, GetTermVectorsIT, MultiSearchIT, OpenCloseIndexIT, SerialDiffIT, IndicesRequestIT, MinIT, DedicatedClusterSnapshotRestoreIT, SimpleSearchIT, FlushIT, FunctionScorePluginIT, FullRollingRestartIT, IndicesOptionsIntegrationIT, ShrinkIndexIT] Completed [83/296] on J0 in 44.25s, 6 tests, 1 failure <<< FAILURES! Suite: org.elasticsearch.action.support.ActiveShardsObserverIT Completed [84/296] on J0 in 2.03s, 6 tests Suite: org.elasticsearch.search.geo.GeoPolygonIT Completed [85/296] on J0 in 0.82s, 2 tests Suite: org.elasticsearch.index.seqno.GlobalCheckpointSyncIT Completed [86/296] on J1 in 20.62s, 2 tests Suite: org.elasticsearch.indices.recovery.IndexPrimaryRelocationIT Completed [87/296] on J0 in 8.93s, 1 test HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:44:22, stalled for 11.6s at: CreateIndexIT.testIndexNameInResponse Suite: org.elasticsearch.action.admin.indices.create.CreateIndexIT Completed [88/296] on J2 in 45.67s, 14 tests Suite: org.elasticsearch.search.aggregations.pipeline.AvgBucketIT Completed [89/296] on J2 in 4.48s, 7 tests Suite: org.elasticsearch.indices.settings.UpdateNumberOfReplicasIT Completed [90/296] on J0 in 10.22s, 5 tests HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:44:32, stalled for 10.3s at: PartitionedRoutingIT.testVariousPartitionSizes Suite: org.elasticsearch.routing.PartitionedRoutingIT Completed [91/296] on J1 in 16.88s, 2 tests Suite: org.elasticsearch.validate.SimpleValidateQueryIT Completed [92/296] on J2 in 4.89s, 12 tests Suite: org.elasticsearch.indices.state.SimpleIndexStateIT Completed [93/296] on J2 in 3.58s, 3 tests Suite: org.elasticsearch.search.aggregations.metrics.TDigestPercentileRanksIT Completed [94/296] on J1 in 5.80s, 21 tests Suite: org.elasticsearch.search.aggregations.AggregationsIntegrationIT Completed [95/296] on J2 in 1.88s, 1 test Suite: org.elasticsearch.ingest.IngestClientIT Completed [96/296] on J1 in 1.46s, 5 tests HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:44:42, stalled for 10.8s at: TopHitsIT (suite) Suite: org.elasticsearch.search.source.SourceFetchingIT Completed [97/296] on J1 in 2.73s, 3 tests Suite: org.elasticsearch.client.transport.TransportClientIT Completed [98/296] on J1 in 1.48s, 4 tests Suite: org.elasticsearch.action.bulk.BulkProcessorIT Completed [99/296] on J2 in 6.37s, 6 tests Suite: org.elasticsearch.search.aggregations.metrics.TopHitsIT Completed [100/296] on J0 in 19.47s, 21 tests Suite: org.elasticsearch.index.mapper.MultiFieldsIntegrationIT Completed [101/296] on J2 in 6.39s, 4 tests Suite: org.elasticsearch.search.SearchWithRejectionsIT Completed [102/296] on J0 in 3.17s, 1 test Suite: org.elasticsearch.action.search.TransportSearchIT Completed [103/296] on J2 in 3.00s, 1 test Suite: org.elasticsearch.search.basic.TransportTwoNodesSearchIT Completed [104/296] on J1 in 10.62s, 10 tests HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:45:06, stalled for 10.7s at: UpdateMappingIntegrationIT.testDynamicUpdates HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:45:12, stalled for 11.9s at: IndicesStoreIntegrationIT.testShardActiveElseWhere HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:45:14, stalled for 10.9s at: ClusterRerouteIT.testDelayWithALargeAmountOfShards HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:45:16, stalled for 20.7s at: UpdateMappingIntegrationIT.testDynamicUpdates Suite: org.elasticsearch.indices.store.IndicesStoreIntegrationIT Completed [105/296] on J1 in 24.95s, 5 tests HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:45:24, stalled for 20.9s at: ClusterRerouteIT.testDelayWithALargeAmountOfShards Suite: org.elasticsearch.search.aggregations.pipeline.BucketSelectorIT Completed [106/296] on J1 in 4.11s, 12 tests Suite: org.elasticsearch.search.aggregations.metrics.HDRPercentilesIT Completed [107/296] on J1 in 3.54s, 18 tests Suite: org.elasticsearch.indices.mapping.UpdateMappingIntegrationIT Completed [108/296] on J2 in 35.51s, 9 tests Suite: org.elasticsearch.search.aggregations.pipeline.ExtendedStatsBucketIT Completed [109/296] on J1 in 5.17s, 9 tests HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:45:34, stalled for 31.1s at: ClusterRerouteIT.testDelayWithALargeAmountOfShards Suite: org.elasticsearch.search.aggregations.pipeline.StatsBucketIT Completed [110/296] on J2 in 3.96s, 7 tests Suite: org.elasticsearch.search.aggregations.bucket.TermsShardMinDocCountIT Completed [111/296] on J2 in 1.18s, 2 tests Suite: org.elasticsearch.search.aggregations.pipeline.DateDerivativeIT Completed [112/296] on J2 in 6.80s, 9 tests HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:45:44, stalled for 41.1s at: ClusterRerouteIT.testDelayWithALargeAmountOfShards Suite: org.elasticsearch.search.aggregations.bucket.GeoDistanceIT Completed [113/296] on J2 in 4.89s, 8 tests Suite: org.elasticsearch.index.mapper.CopyToMapperIntegrationIT Completed [114/296] on J2 in 2.22s, 2 tests Suite: org.elasticsearch.cluster.allocation.ClusterRerouteIT Completed [115/296] on J0 in 56.61s, 8 tests HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:45:52, stalled for 10.7s at: DelayedAllocationIT.testDelayedAllocationTimesOut Suite: org.elasticsearch.search.basic.SearchWhileCreatingIndexIT Completed [116/296] on J2 in 3.39s, 4 tests Suite: org.elasticsearch.action.admin.cluster.stats.ClusterStatsIT Completed [117/296] on J2 in 3.59s, 5 tests Suite: org.elasticsearch.search.aggregations.metrics.CardinalityIT Completed [118/296] on J2 in 1.97s, 19 tests Suite: org.elasticsearch.indexlifecycle.IndexLifecycleActionIT Completed [119/296] on J0 in 8.92s, 1 test Suite: org.elasticsearch.action.admin.indices.delete.DeleteIndexBlocksIT Completed [120/296] on J2 in 2.44s, 3 tests Suite: org.elasticsearch.cluster.routing.DelayedAllocationIT Completed [121/296] on J1 in 30.43s, 5 tests Suite: org.elasticsearch.gateway.QuorumGatewayIT Completed [122/296] on J1 in 5.03s, 1 test Suite: org.elasticsearch.update.UpdateIT Completed [123/296] on J0 in 12.25s, 12 tests Suite: org.elasticsearch.action.bulk.BulkProcessorRetryIT Completed [124/296] on J0 in 4.77s, 2 tests Suite: org.elasticsearch.search.SearchCancellationIT Completed [125/296] on J1 in 10.09s, 4 tests Suite: org.elasticsearch.search.aggregations.metrics.AvgIT Completed [126/296] on J1 in 1.88s, 17 tests Suite: org.elasticsearch.indices.analyze.AnalyzeActionIT Completed [127/296] on J0 in 4.85s, 15 tests Suite: org.elasticsearch.index.suggest.stats.SuggestStatsIT Completed [128/296] on J0 in 4.48s, 1 test Suite: org.elasticsearch.action.admin.indices.get.GetIndexIT Completed [129/296] on J0 in 2.05s, 9 tests Suite: org.elasticsearch.search.aggregations.pipeline.MinBucketIT Completed [130/296] on J0 in 3.18s, 7 tests Suite: org.elasticsearch.search.sort.FieldSortIT Completed [131/296] on J2 in 35.92s, 20 tests Suite: org.elasticsearch.search.aggregations.EquivalenceIT Completed [132/296] on J1 in 16.69s, 6 tests Suite: org.elasticsearch.index.search.MatchQueryIT Completed [133/296] on J0 in 7.08s, 6 tests Suite: org.elasticsearch.indices.template.IndexTemplateBlocksIT Completed [134/296] on J2 in 0.82s, 1 test Suite: org.elasticsearch.search.aggregations.bucket.FiltersIT Completed [135/296] on J1 in 2.17s, 12 tests Suite: org.elasticsearch.search.aggregations.metrics.MaxIT Completed [136/296] on J2 in 2.08s, 17 tests Suite: org.elasticsearch.routing.AliasRoutingIT Completed [137/296] on J0 in 10.17s, 6 tests Suite: org.elasticsearch.search.aggregations.metrics.SumIT Completed [138/296] on J0 in 2.10s, 17 tests Suite: org.elasticsearch.cluster.SpecificMasterNodesIT Completed [139/296] on J1 in 10.89s, 3 tests Suite: org.elasticsearch.persistent.PersistentTasksExecutorIT Completed [140/296] on J1 in 1.84s, 5 tests Suite: org.elasticsearch.search.aggregations.bucket.MissingIT Completed [141/296] on J1 in 0.91s, 5 tests Suite: org.elasticsearch.cluster.NoMasterNodeIT Completed [142/296] on J1 in 5.53s, 2 tests HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:47:00, stalled for 10.1s at: SimpleIndexTemplateIT.testThatGetIndexTemplatesWorks Suite: org.elasticsearch.get.GetActionIT Completed [143/296] on J1 in 9.40s, 13 tests Suite: org.elasticsearch.search.aggregations.bucket.RangeIT Completed [144/296] on J1 in 3.68s, 17 tests Suite: org.elasticsearch.indices.template.SimpleIndexTemplateIT Completed [145/296] on J0 in 31.66s, 22 tests Suite: org.elasticsearch.search.source.MetadataFetchingIT Completed [146/296] on J0 in 1.49s, 3 tests Suite: org.elasticsearch.cluster.settings.ClusterSettingsIT Completed [147/296] on J1 in 11.67s, 9 tests Suite: org.elasticsearch.client.transport.NodeDisconnectIT Completed [148/296] on J0 in 1.97s, 2 tests Suite: org.elasticsearch.cluster.ClusterInfoServiceIT Completed [149/296] on J1 in 3.91s, 2 tests Suite: org.elasticsearch.search.basic.TransportSearchFailuresIT Completed [150/296] on J0 in 2.81s, 1 test Suite: org.elasticsearch.search.suggest.ContextCompletionSuggestSearchIT IGNOR/A 0.00s J1 | ContextCompletionSuggestSearchIT.testMultiContextBoosting > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=multiple context boosting is broken, as a suggestion, contexts pair is treated as (num(context) entries))) Completed [151/296] on J1 in 15.89s, 18 tests, 1 skipped HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:47:53, stalled for 11.1s at: IndexStatsIT.testConcurrentIndexingAndStatsRequests Suite: org.elasticsearch.document.AliasedIndexDocumentActionsIT Completed [152/296] on J1 in 10.58s, 2 tests Suite: org.elasticsearch.indices.stats.IndexStatsIT Completed [153/296] on J0 in 34.49s, 18 tests Suite: org.elasticsearch.versioning.SimpleVersioningIT Completed [154/296] on J1 in 9.01s, 11 tests Suite: org.elasticsearch.search.scroll.SearchScrollWithFailingNodesIT Completed [155/296] on J1 in 0.97s, 1 test Suite: org.elasticsearch.action.admin.indices.flush.FlushBlocksIT Completed [156/296] on J1 in 1.63s, 1 test Suite: org.elasticsearch.search.aggregations.pipeline.BucketScriptIT Completed [157/296] on J0 in 5.11s, 10 tests Suite: org.elasticsearch.search.aggregations.metrics.HDRPercentileRanksIT Completed [158/296] on J0 in 2.77s, 21 tests Suite: org.elasticsearch.snapshots.SharedClusterSnapshotRestoreIT Completed [159/296] on J2 in 91.30s, 50 tests Suite: org.elasticsearch.search.aggregations.pipeline.DerivativeIT Completed [160/296] on J0 in 1.79s, 14 tests Suite: org.elasticsearch.cluster.metadata.TemplateUpgradeServiceIT Completed [161/296] on J0 in 0.48s, 1 test Suite: org.elasticsearch.gateway.MetaDataWriteDataNodesIT Completed [162/296] on J0 in 1.44s, 3 tests Suite: org.elasticsearch.script.StoredScriptsIT Completed [163/296] on J0 in 0.22s, 2 tests HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:48:17, stalled for 10.8s at: CircuitBreakerServiceIT.testMemoryBreaker Suite: org.elasticsearch.search.aggregations.bucket.terms.StringTermsIT Completed [164/296] on J0 in 3.80s, 35 tests Suite: org.elasticsearch.search.aggregations.CombiIT Completed [165/296] on J0 in 0.70s, 2 tests Suite: org.elasticsearch.search.aggregations.bucket.FilterIT Completed [166/296] on J0 in 2.87s, 6 tests HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:48:23, stalled for 11.2s at: SingleNodeDiscoveryIT.testDoesNotRespondToZenPings Suite: org.elasticsearch.cluster.SimpleDataNodesIT Completed [167/296] on J0 in 2.43s, 1 test Suite: org.elasticsearch.persistent.PersistentTasksExecutorFullRestartIT Completed [168/296] on J0 in 3.74s, 1 test Suite: org.elasticsearch.indices.memory.breaker.CircuitBreakerServiceIT Completed [169/296] on J1 in 23.54s, 7 tests Suite: org.elasticsearch.search.sort.GeoDistanceSortBuilderIT Completed [170/296] on J1 in 2.32s, 5 tests HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:48:35, stalled for 10.2s at: SingleNodeDiscoveryIT.testSingleNodesDoNotDiscoverEachOther Suite: org.elasticsearch.discovery.single.SingleNodeDiscoveryIT Completed [171/296] on J2 in 23.54s, 2 tests Suite: org.elasticsearch.search.scriptfilter.ScriptQuerySearchIT Completed [172/296] on J1 in 10.55s, 2 tests Suite: org.elasticsearch.search.aggregations.bucket.MinDocCountIT Completed [173/296] on J0 in 14.81s, 36 tests Suite: org.elasticsearch.search.morelikethis.MoreLikeThisIT Completed [174/296] on J2 in 10.82s, 17 tests Suite: org.elasticsearch.search.searchafter.SearchAfterIT Completed [175/296] on J2 in 4.53s, 3 tests HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:48:53, stalled for 10.3s at: GeoBoundsIT (suite) Suite: org.elasticsearch.search.aggregations.bucket.ReverseNestedIT Completed [176/296] on J2 in 2.52s, 6 tests Suite: org.elasticsearch.search.sort.SimpleSortIT Completed [177/296] on J2 in 4.87s, 4 tests Suite: org.elasticsearch.search.aggregations.metrics.GeoBoundsIT Completed [178/296] on J1 in 17.38s, 10 tests Suite: org.elasticsearch.indices.IndicesLifecycleListenerIT Completed [179/296] on J2 in 3.04s, 3 tests Suite: org.elasticsearch.cluster.MinimumMasterNodesIT Completed [180/296] on J0 in 19.01s, 5 tests Suite: org.elasticsearch.search.aggregations.bucket.HistogramIT Completed [181/296] on J2 in 4.10s, 35 tests Suite: org.elasticsearch.search.fetch.subphase.MatchedQueriesIT Completed [182/296] on J1 in 7.17s, 10 tests Suite: org.elasticsearch.index.store.ExceptionRetryIT Completed [183/296] on J2 in 2.54s, 1 test Suite: org.elasticsearch.search.fetch.FetchSubPhasePluginIT Completed [184/296] on J2 in 1.38s, 1 test Suite: org.elasticsearch.client.transport.TransportClientRetryIT Completed [185/296] on J2 in 0.12s, 1 test Suite: org.elasticsearch.search.aggregations.FiltersAggsRewriteIT Completed [186/296] on J2 in 0.20s, 1 test Suite: org.elasticsearch.search.functionscore.DecayFunctionScoreIT Completed [187/296] on J2 in 6.73s, 14 tests Suite: org.elasticsearch.cluster.routing.allocation.decider.UpdateShardAllocationSettingsIT Completed [188/296] on J2 in 1.38s, 2 tests Suite: org.elasticsearch.search.aggregations.bucket.ShardReduceIT Completed [189/296] on J1 in 13.20s, 14 tests Suite: org.elasticsearch.gateway.RecoveryFromGatewayIT Completed [190/296] on J0 in 28.83s, 8 tests Suite: org.elasticsearch.search.SearchTimeoutIT Completed [191/296] on J2 in 14.19s, 2 tests Suite: org.elasticsearch.action.admin.indices.shards.IndicesShardStoreRequestIT Completed [192/296] on J1 in 12.60s, 4 tests Suite: org.elasticsearch.indices.state.CloseIndexDisableCloseAllIT Completed [193/296] on J2 in 2.05s, 1 test Suite: org.elasticsearch.search.aggregations.bucket.NestedIT Completed [194/296] on J0 in 3.71s, 10 tests Suite: org.elasticsearch.snapshots.BlobStoreFormatIT Completed [195/296] on J2 in 0.83s, 5 tests Suite: org.elasticsearch.action.admin.indices.cache.clear.ClearIndicesCacheBlocksIT Completed [196/296] on J1 in 2.20s, 1 test Suite: org.elasticsearch.search.functionscore.FunctionScoreFieldValueIT Completed [197/296] on J1 in 1.33s, 1 test Suite: org.elasticsearch.cluster.service.ClusterServiceIT Completed [198/296] on J0 in 2.43s, 5 tests Suite: org.elasticsearch.search.aggregations.MissingValueIT Completed [199/296] on J1 in 0.89s, 13 tests Suite: org.elasticsearch.search.aggregations.bucket.LongTermsIT Completed [200/296] on J2 in 3.50s, 29 tests Suite: org.elasticsearch.search.basic.SearchWithRandomExceptionsIT Completed [201/296] on J1 in 1.08s, 1 test Suite: org.elasticsearch.action.RejectionActionIT Completed [202/296] on J1 in 6.31s, 1 test Suite: org.elasticsearch.search.geo.GeoFilterIT Completed [203/296] on J2 in 8.03s, 4 tests HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:49:47, stalled for 10.2s at: ClusterHealthIT.testHealth Suite: org.elasticsearch.cluster.ClusterHealthIT Completed [204/296] on J0 in 18.15s, 3 tests HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:49:57, stalled for 10.3s at: RecoverAfterNodesIT.testRecoverAfterNodes HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:49:57, stalled for 11.9s at: HotThreadsIT.testHotThreadsDontFail Suite: org.elasticsearch.gateway.RecoverAfterNodesIT Completed [205/296] on J2 in 14.01s, 3 tests Suite: org.elasticsearch.search.stats.SearchStatsIT Completed [206/296] on J2 in 6.47s, 2 tests Suite: org.elasticsearch.indices.memory.breaker.RandomExceptionCircuitBreakerIT Completed [207/296] on J2 in 2.47s, 1 test Suite: org.elasticsearch.index.shard.IndexShardIT Completed [208/296] on J2 in 6.51s, 15 tests Suite: org.elasticsearch.search.suggest.CompletionSuggestSearchIT Completed [209/296] on J0 in 20.94s, 38 tests Suite: org.elasticsearch.action.admin.HotThreadsIT Completed [210/296] on J1 in 30.83s, 3 tests Suite: org.elasticsearch.search.basic.SearchWithRandomIOExceptionsIT Completed [211/296] on J2 in 2.25s, 1 test Suite: org.elasticsearch.discovery.zen.ZenDiscoveryIT Completed [212/296] on J0 in 10.46s, 5 tests Suite: org.elasticsearch.versioning.ConcurrentDocumentOperationIT Completed [213/296] on J0 in 0.43s, 1 test Suite: org.elasticsearch.search.scroll.SearchScrollIT Completed [214/296] on J0 in 3.38s, 13 tests Suite: org.elasticsearch.action.admin.indices.exists.IndicesExistsIT Completed [215/296] on J0 in 0.09s, 1 test Suite: org.elasticsearch.indices.memory.breaker.CircuitBreakerNoopIT Completed [216/296] on J0 in 0.60s, 2 tests Suite: org.elasticsearch.search.sort.GeoDistanceIT Completed [217/296] on J0 in 2.16s, 4 tests Suite: org.elasticsearch.action.admin.cluster.repositories.RepositoryBlocksIT Completed [218/296] on J2 in 18.78s, 4 tests Suite: org.elasticsearch.search.query.SimpleQueryStringIT Completed [219/296] on J0 in 8.39s, 17 tests Suite: org.elasticsearch.action.admin.cluster.allocation.ClusterAllocationExplainIT Completed [220/296] on J1 in 27.15s, 10 tests Suite: org.elasticsearch.action.admin.indices.forcemerge.ForceMergeBlocksIT Completed [221/296] on J1 in 0.95s, 1 test Suite: org.elasticsearch.search.aggregations.metrics.ExtendedStatsIT Completed [222/296] on J0 in 3.83s, 20 tests Suite: org.elasticsearch.search.fetch.subphase.InnerHitsIT Completed [223/296] on J0 in 4.23s, 10 tests Suite: org.elasticsearch.search.aggregations.bucket.SamplerIT Completed [224/296] on J0 in 0.40s, 4 tests Suite: org.elasticsearch.action.admin.indices.segments.IndicesSegmentsBlocksIT Completed [225/296] on J0 in 1.01s, 1 test Suite: org.elasticsearch.action.admin.cluster.tasks.PendingTasksBlocksIT Completed [226/296] on J0 in 1.82s, 1 test Suite: org.elasticsearch.action.admin.indices.stats.IndicesStatsBlocksIT Completed [227/296] on J1 in 9.31s, 1 test Suite: org.elasticsearch.action.bulk.BulkIntegrationIT Completed [228/296] on J1 in 0.69s, 1 test Suite: org.elasticsearch.search.functionscore.ExplainableScriptIT Completed [229/296] on J0 in 1.99s, 1 test Suite: org.elasticsearch.action.bulk.BulkProcessorClusterSettingsIT Completed [230/296] on J1 in 0.69s, 1 test Suite: org.elasticsearch.bwcompat.RecoveryWithUnsupportedIndicesIT Completed [231/296] on J1 in 0.87s, 1 test Suite: org.elasticsearch.search.aggregations.MetaDataIT Completed [232/296] on J0 in 1.33s, 1 test Suite: org.elasticsearch.action.admin.cluster.snapshots.SnapshotBlocksIT Completed [233/296] on J2 in 22.70s, 6 tests Suite: org.elasticsearch.client.documentation.IndicesDocumentationIT Completed [234/296] on J2 in 0.84s, 1 test Suite: org.elasticsearch.search.basic.SearchWhileRelocatingIT Completed [235/296] on J0 in 4.68s, 1 test Suite: org.elasticsearch.cluster.ack.AckClusterUpdateSettingsIT Completed [236/296] on J2 in 2.50s, 3 tests Suite: org.elasticsearch.bwcompat.RestoreBackwardsCompatIT Completed [237/296] on J1 in 7.93s, 1 test Suite: org.elasticsearch.cluster.allocation.SimpleAllocationIT Completed [238/296] on J2 in 1.97s, 1 test Suite: org.elasticsearch.cluster.routing.allocation.ShardStateIT Completed [239/296] on J2 in 1.49s, 1 test Suite: org.elasticsearch.cluster.ack.AckIT Completed [240/296] on J0 in 13.96s, 15 tests HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:51:15, stalled for 10.6s at: PrimaryAllocationIT.testDoNotAllowStaleReplicasToBePromotedToPrimary Suite: org.elasticsearch.cluster.routing.allocation.decider.MockDiskUsagesIT Completed [241/296] on J2 in 8.89s, 1 test HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:51:27, stalled for 11.8s at: MasterDisruptionIT.testNodesFDAfterMasterReelection HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:51:27, stalled for 11.3s at: SnapshotDisruptionIT.testDisruptionOnSnapshotInitialization HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:51:37, stalled for 21.3s at: SnapshotDisruptionIT.testDisruptionOnSnapshotInitialization HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:51:47, stalled for 31.3s at: SnapshotDisruptionIT.testDisruptionOnSnapshotInitialization HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:51:49, stalled for 11.0s at: MasterDisruptionIT.testStaleMasterNotHijackingMajority HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:51:55, stalled for 10.5s at: PrimaryAllocationIT.testNotWaitForQuorumCopies HEARTBEAT J2 PID(9936@SLAVE-0633DD87F): 2018-05-05T08:51:57, stalled for 41.3s at: SnapshotDisruptionIT.testDisruptionOnSnapshotInitialization Suite: org.elasticsearch.discovery.SnapshotDisruptionIT Completed [242/296] on J2 in 43.22s, 1 test Suite: org.elasticsearch.document.DocumentActionsIT Completed [243/296] on J2 in 1.18s, 2 tests Suite: org.elasticsearch.document.ShardInfoIT Completed [244/296] on J2 in 0.63s, 4 tests Suite: org.elasticsearch.explain.ExplainActionIT Completed [245/296] on J2 in 3.80s, 7 tests HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:52:05, stalled for 11.5s at: MasterDisruptionIT.testIsolateMasterAndVerifyClusterStateConsensus Suite: org.elasticsearch.index.IndexSortIT Completed [246/296] on J2 in 1.41s, 2 tests Suite: org.elasticsearch.index.SettingsListenerIT Completed [247/296] on J2 in 0.42s, 1 test Suite: org.elasticsearch.index.WaitUntilRefreshIT Completed [248/296] on J2 in 3.34s, 5 tests Suite: org.elasticsearch.index.mapper.DynamicMappingIT Completed [249/296] on J2 in 3.92s, 4 tests Suite: org.elasticsearch.index.mapper.ExternalValuesMapperIntegrationIT Completed [250/296] on J2 in 1.03s, 3 tests Suite: org.elasticsearch.index.query.plugin.CustomQueryParserIT Completed [252/296] on J2 in 1.66s, 4 tests HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:52:21, stalled for 11.7s at: PrimaryAllocationIT.testDoNotRemoveAllocationIdOnNodeLeave HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:52:23, stalled for 10.2s at: MasterDisruptionIT.testMasterNodeGCs Suite: org.elasticsearch.indices.exists.types.TypesExistsIT Completed [253/296] on J2 in 9.49s, 2 tests Suite: org.elasticsearch.indices.mapping.SimpleGetFieldMappingsIT Completed [254/296] on J2 in 2.95s, 5 tests Suite: org.elasticsearch.indices.mapping.SimpleGetMappingsIT Completed [255/296] on J2 in 0.85s, 3 tests Suite: org.elasticsearch.indices.settings.GetSettingsBlocksIT Completed [256/296] on J2 in 0.95s, 1 test Suite: org.elasticsearch.ingest.IngestProcessorNotInstalledOnAllNodesIT Completed [257/296] on J2 in 2.59s, 3 tests HEARTBEAT J1 PID(2948@SLAVE-0633DD87F): 2018-05-05T08:52:35, stalled for 10.2s at: PrimaryAllocationIT.testRemoveAllocationIdOnWriteAfterNodeLeave Suite: org.elasticsearch.mget.SimpleMgetIT Completed [258/296] on J2 in 3.54s, 5 tests Suite: org.elasticsearch.nodesinfo.SimpleNodesInfoIT Completed [259/296] on J2 in 0.56s, 3 tests HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:52:43, stalled for 11.6s at: MasterDisruptionIT.testVerifyApiBlocksDuringPartition Suite: org.elasticsearch.cluster.routing.PrimaryAllocationIT Completed [260/296] on J1 in 99.79s, 9 tests Suite: org.elasticsearch.operateAllIndices.DestructiveOperationsIT Completed [261/296] on J2 in 4.63s, 6 tests Suite: org.elasticsearch.recovery.SimpleRecoveryIT Completed [262/296] on J2 in 0.89s, 1 test Suite: org.elasticsearch.recovery.TruncatedRecoveryIT Completed [263/296] on J2 in 1.10s, 1 test Suite: org.elasticsearch.routing.AliasResolveRoutingIT Completed [264/296] on J2 in 2.07s, 3 tests Suite: org.elasticsearch.search.aggregations.bucket.BooleanTermsIT Completed [265/296] on J2 in 2.77s, 3 tests Suite: org.elasticsearch.recovery.RelocationIT Completed [266/296] on J1 in 11.06s, 5 tests HEARTBEAT J0 PID(6640@SLAVE-0633DD87F): 2018-05-05T08:52:57, stalled for 11.3s at: MasterDisruptionIT.testFailWithMinimumMasterNodesConfigured Suite: org.elasticsearch.discovery.MasterDisruptionIT Completed [267/296] on J0 in 102.18s, 7 tests Suite: org.elasticsearch.search.aggregations.bucket.DateRangeIT Completed [268/296] on J1 in 2.93s, 19 tests Suite: org.elasticsearch.search.aggregations.bucket.DiversifiedSamplerIT Completed [269/296] on J0 in 3.66s, 6 tests Suite: org.elasticsearch.search.aggregations.bucket.GlobalIT Completed [270/296] on J0 in 1.82s, 2 tests Suite: org.elasticsearch.search.aggregations.bucket.GeoHashGridIT Completed [271/296] on J1 in 5.96s, 8 tests Suite: org.elasticsearch.search.aggregations.bucket.DateHistogramIT Completed [272/296] on J2 in 12.30s, 40 tests Suite: org.elasticsearch.search.aggregations.bucket.NaNSortingIT Completed [273/296] on J2 in 0.81s, 4 tests Suite: org.elasticsearch.search.aggregations.bucket.IpRangeIT Completed [274/296] on J0 in 1.99s, 8 tests Suite: org.elasticsearch.search.aggregations.bucket.IpTermsIT Completed [275/296] on J1 in 1.55s, 3 tests Suite: org.elasticsearch.search.aggregations.bucket.SignificantTermsSignificanceScoreIT Completed [276/296] on J0 in 3.84s, 8 tests Suite: org.elasticsearch.search.aggregations.bucket.TermsDocCountErrorIT Completed [277/296] on J1 in 4.25s, 25 tests Suite: org.elasticsearch.search.aggregations.metrics.StatsIT Completed [278/296] on J0 in 3.35s, 18 tests Suite: org.elasticsearch.search.aggregations.metrics.TDigestPercentilesIT Completed [279/296] on J1 in 5.39s, 18 tests Suite: org.elasticsearch.search.aggregations.bucket.ShardSizeTermsIT Completed [280/296] on J2 in 12.20s, 15 tests Suite: org.elasticsearch.search.aggregations.pipeline.bucketsort.BucketSortIT Completed [281/296] on J2 in 1.70s, 13 tests Suite: org.elasticsearch.search.aggregations.pipeline.moving.avg.MovAvgIT IGNOR/A 0.01s J2 | MovAvgIT.testBadModelParams > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testSizeZeroWindow > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testPredictNegativeKeysAtStart > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testLinearSingleValuedField > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testSimpleSingleValuedField > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testNoBucketsInHistogram > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testBadParent > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testHoltSingleValuedField > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testPredictWithNonEmptyBuckets > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testCheckIfTunableCanBeMinimized > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testNegativeWindow > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testCheckIfNonTunableCanBeMinimized > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testHoltWintersMinimization > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testNegativePrediction > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testNoBucketsInHistogramWithPredict > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testEwmaSingleValuedField > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testZeroPrediction > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testMinimizeNotEnoughData > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testTwoMovAvgsWithPredictions > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testHoltWintersValuedField > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) IGNOR/A 0.00s J2 | MovAvgIT.testHoltWintersNotEnoughData > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://github.com/elastic/elasticsearch/issues/29456)) Completed [282/296] on J2 in 0.01s, 21 tests, 21 skipped Suite: org.elasticsearch.search.fetch.subphase.highlight.CustomHighlighterSearchIT Completed [283/296] on J2 in 0.78s, 4 tests Suite: org.elasticsearch.search.aggregations.pipeline.MaxBucketIT Completed [284/296] on J0 in 9.96s, 8 tests Suite: org.elasticsearch.search.preference.SearchPreferenceIT Completed [285/296] on J2 in 2.91s, 5 tests Suite: org.elasticsearch.search.aggregations.pipeline.SumBucketIT Completed [286/296] on J1 in 11.53s, 7 tests Suite: org.elasticsearch.search.query.MultiMatchQueryIT Completed [287/296] on J1 in 5.15s, 7 tests Suite: org.elasticsearch.search.profile.aggregation.AggregationProfilerIT Completed [288/296] on J0 in 11.37s, 6 tests Suite: org.elasticsearch.search.profile.query.QueryProfilerIT Completed [289/296] on J2 in 15.20s, 11 tests Suite: org.elasticsearch.search.suggest.CustomSuggesterSearchIT Completed [290/296] on J2 in 0.82s, 1 test Suite: org.elasticsearch.search.scroll.DuelScrollIT Completed [291/296] on J0 in 5.58s, 2 tests Suite: org.elasticsearch.search.query.QueryStringIT Completed [292/296] on J1 in 7.60s, 11 tests Suite: org.elasticsearch.snapshots.RepositoriesIT Completed [293/296] on J0 in 0.61s, 4 tests Suite: org.elasticsearch.snapshots.SnapshotShardsServiceIT Completed [294/296] on J1 in 0.93s, 1 test Suite: org.elasticsearch.threadpool.SimpleThreadPoolIT Completed [295/296] on J0 in 0.56s, 1 test Suite: org.elasticsearch.snapshots.MinThreadsSnapshotRestoreIT Completed [296/296] on J2 in 2.61s, 3 tests > Task :server:integTest FAILED Tests with failures: - org.elasticsearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm - org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testCreateShrinkIndexToN Slow Tests Summary: 102.18s | org.elasticsearch.discovery.MasterDisruptionIT 99.79s | org.elasticsearch.cluster.routing.PrimaryAllocationIT 91.30s | org.elasticsearch.snapshots.SharedClusterSnapshotRestoreIT 73.71s | org.elasticsearch.search.fetch.subphase.highlight.HighlighterSearchIT 72.02s | org.elasticsearch.discovery.DiscoveryDisruptionIT [ant:junit4] JVM J0: 0.86 .. 1018.58 = 1017.72s [ant:junit4] JVM J1: 0.63 .. 1018.56 = 1017.93s [ant:junit4] JVM J2: 0.86 .. 1019.69 = 1018.83s [ant:junit4] Execution time total: 16 minutes 59 seconds [ant:junit4] Tests summary: 296 suites (1 ignored), 2201 tests, 2 failures, 23 ignored (23 assumptions) :server:integTest (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 17 mins 0.087 secs. :plugins:analysis-icu:integTestCluster#stop (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :plugins:analysis-icu:integTestCluster#stop Task ':plugins:analysis-icu:integTestCluster#stop' is not up-to-date because: Task has not declared any outputs despite executing actions. Shutting down external node with pid 9132 Starting process 'command 'Taskkill''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\analysis-icu Command: Taskkill /PID 9132 /F Successfully started process 'command 'Taskkill'' SUCCESS: The process with PID 9132 has been terminated. :plugins:analysis-icu:integTestCluster#stop (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.063 secs. :docs:integTestCluster#stop (Thread[Task worker for ':' Thread 3,5,main]) started. > Task :docs:integTestCluster#stop Task ':docs:integTestCluster#stop' is not up-to-date because: Task has not declared any outputs despite executing actions. FAILURE: Build failed with an exception. Shutting down external node with pid 12088 Starting process 'command 'Taskkill''. Working directory: C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\docs Command: Taskkill /PID 12088 /F * What went wrong: Successfully started process 'command 'Taskkill'' Execution failed for task ':server:integTest'. SUCCESS: The process with PID 12088 has been terminated. > There were test failures: 296 suites (1 ignored), 2201 tests, 2 failures, 23 ignored (23 assumptions) [seed: 734B52BB31887C24] * Try: :docs:integTestCluster#stop (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 0.109 secs. Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights. Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. See https://docs.gradle.org/4.7/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 33m 27s 515 actionable tasks: 513 executed, 2 up-to-date * Get more help at https://help.gradle.org Stopped 1 worker daemon(s). runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<< runbld>>> DURATION: 2072817ms runbld>>> STDOUT: 4230641 bytes runbld>>> STDERR: 429 bytes runbld>>> WRAPPED PROCESS: FAILURE (1) runbld>>> Searching for build metadata in C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility runbld>>> Storing build metadata: WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by clojure.lang.Reflector (file:/C:/Program%20Files/infra/bin/runbld) to method com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(org.xml.sax.InputSource,org.xml.sax.helpers.DefaultHandler) WARNING: Please consider reporting this to the maintainers of clojure.lang.Reflector WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release runbld>>> FAILURES: 2 runbld>>> BUILD: https://5086a1f436ee16623a447bdf25881bbc.us-east-1.aws.found.io:9243/build-1498188341992/t/20180505081907-74967D11 runbld>>> MAILING: build-elasticsearch@e***.co runbld>>> Mail sent. Output: {:code 0, :error :SUCCESS, :message "messages sent"} runbld>>> NOTIFYING SLACK C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility>exit 1 Build step 'Execute Windows batch command' marked build as failure Archiving artifacts Sending e-mails to: infra-root+build@elastic.co Recording test results Finished: FAILURE