Skip to content

Commit b6227db

Browse files
committed
Merge branch '6.x' into ccr-6.x
* 6.x: (32 commits) Use a dedicated ThreadGroup in rest sniffer (#26897) Fire global checkpoint sync under system context Update by Query is modified to accept short `script` parameter. (#26841) Cat shards bytes (#26952) Adding unreleased 5.6.4 version number to Version.java Rename TCPTransportTests to TcpTransportTests (#26954) Fix NPE for /_cat/indices when no primary shard (#26953) [DOCS] Fixed indentation of the definition list. Check for closed connection while opening Clarify systemd overrides [DOCS] Plugin Installation for Windows (#21671) Painless: add tests for cached boxing (#24163) Don't detect source's XContentType in DocumentParser.parseDocument() (#26880) Return List instead of an array from settings (#26903) Emit deprecation warning for variable size predictor Fix handling of paths containing parentheses Deprecate variable-size receive predictor Add Homebrew instructions to getting started ingest: Fix bug that prevent date_index_name processor from accepting timestamps specified as a json number Scripting: Fix expressions to temporarily support filter scripts (#26824) ...
2 parents db49494 + 22811b5 commit b6227db

File tree

120 files changed

+1095
-647
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

120 files changed

+1095
-647
lines changed

CONTRIBUTING.md

+5
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,11 @@ If you have a bugfix or new feature that you would like to contribute to Elastic
3838

3939
We enjoy working with contributors to get their code accepted. There are many approaches to fixing a problem and it is important to find the best approach before writing too much code.
4040

41+
Note that it is unlikely the project will merge refactors for the sake of refactoring. These
42+
types of pull requests have a high cost to maintainers in reviewing and testing with little
43+
to no tangible benefit. This especially includes changes generated by tools. For example,
44+
converting all generic interface instances to use the diamond operator.
45+
4146
The process for contributing to any of the [Elastic repositories](https://github.com/elastic/) is similar. Details for individual projects can be found below.
4247

4348
### Fork and clone the repository

TESTING.asciidoc

+13-11
Original file line numberDiff line numberDiff line change
@@ -472,28 +472,30 @@ is tested depends on the branch. On master, this will test against the current
472472
stable branch. On the stable branch, it will test against the latest release
473473
branch. Finally, on a release branch, it will test against the most recent release.
474474

475-
=== BWC Testing against a specific branch
475+
=== BWC Testing against a specific remote/branch
476476

477477
Sometimes a backward compatibility change spans two versions. A common case is a new functionality
478478
that needs a BWC bridge in and an unreleased versioned of a release branch (for example, 5.x).
479-
To test the changes, you can instruct gradle to build the BWC version from a local branch instead of
480-
pulling the release branch from GitHub. You do so using the `tests.bwc.refspec` system property:
479+
To test the changes, you can instruct gradle to build the BWC version from a another remote/branch combination instead of
480+
pulling the release branch from GitHub. You do so using the `tests.bwc.remote` and `tests.bwc.refspec` system properties:
481481

482482
-------------------------------------------------
483-
gradle check -Dtests.bwc.refspec=origin/index_req_bwc_5.x
483+
gradle check -Dtests.bwc.remote=${remote} -Dtests.bwc.refspec=index_req_bwc_5.x
484484
-------------------------------------------------
485485

486-
The branch needs to be available on the local clone that the BWC makes of the repository you run the
487-
tests from. Using the `origin` remote is a handy trick to make sure that a branch is available
488-
and is up to date in the case of multiple runs.
486+
The branch needs to be available on the remote that the BWC makes of the
487+
repository you run the tests from. Using the remote is a handy trick to make
488+
sure that a branch is available and is up to date in the case of multiple runs.
489489

490490
Example:
491491

492-
Say you need to make a change to `master` and have a BWC layer in `5.x`. You will need to:
493-
. Create a branch called `index_req_change` off `master`. This will contain your change.
492+
Say you need to make a change to `master` and have a BWC layer in `5.x`. You
493+
will need to:
494+
. Create a branch called `index_req_change` off your remote `${remote}`. This
495+
will contain your change.
494496
. Create a branch called `index_req_bwc_5.x` off `5.x`. This will contain your bwc layer.
495-
. If not running the tests locally, push both branches to your remote repository.
496-
. Run the tests with `gradle check -Dtests.bwc.refspec=origin/index_req_bwc_5.x`
497+
. Push both branches to your remote repository.
498+
. Run the tests with `gradle check -Dtests.bwc.remote=${remote} -Dtests.bwc.refspec=index_req_bwc_5.x`.
497499

498500
== Coverage analysis
499501

buildSrc/src/main/groovy/org/elasticsearch/gradle/test/ClusterConfiguration.groovy

+3-5
Original file line numberDiff line numberDiff line change
@@ -63,13 +63,11 @@ class ClusterConfiguration {
6363
boolean debug = false
6464

6565
/**
66-
* if <code>true</code> each node will be configured with <tt>discovery.zen.minimum_master_nodes</tt> set
67-
* to the total number of nodes in the cluster. This will also cause that each node has `0s` state recovery
68-
* timeout which can lead to issues if for instance an existing clusterstate is expected to be recovered
69-
* before any tests start
66+
* Configuration of the setting <tt>discovery.zen.minimum_master_nodes</tt> on the nodes.
67+
* In case of more than one node, this defaults to (number of nodes / 2) + 1
7068
*/
7169
@Input
72-
boolean useMinimumMasterNodes = true
70+
Closure<Integer> minimumMasterNodes = { getNumNodes() > 1 ? getNumNodes().intdiv(2) + 1 : -1 }
7371

7472
@Input
7573
String jvmArgs = "-Xms" + System.getProperty('tests.heap.size', '512m') +

buildSrc/src/main/groovy/org/elasticsearch/gradle/test/ClusterFormationTasks.groovy

+8-7
Original file line numberDiff line numberDiff line change
@@ -311,13 +311,14 @@ class ClusterFormationTasks {
311311
// Define a node attribute so we can test that it exists
312312
'node.attr.testattr' : 'test'
313313
]
314-
// we set min master nodes to the total number of nodes in the cluster and
315-
// basically skip initial state recovery to allow the cluster to form using a realistic master election
316-
// this means all nodes must be up, join the seed node and do a master election. This will also allow new and
317-
// old nodes in the BWC case to become the master
318-
if (node.config.useMinimumMasterNodes && node.config.numNodes > 1) {
319-
esConfig['discovery.zen.minimum_master_nodes'] = node.config.numNodes
320-
esConfig['discovery.initial_state_timeout'] = '0s' // don't wait for state.. just start up quickly
314+
int minimumMasterNodes = node.config.minimumMasterNodes.call()
315+
if (minimumMasterNodes > 0) {
316+
esConfig['discovery.zen.minimum_master_nodes'] = minimumMasterNodes
317+
}
318+
if (node.config.numNodes > 1) {
319+
// don't wait for state.. just start up quickly
320+
// this will also allow new and old nodes in the BWC case to become the master
321+
esConfig['discovery.initial_state_timeout'] = '0s'
321322
}
322323
esConfig['node.max_local_storage_nodes'] = node.config.numNodes
323324
esConfig['http.port'] = node.config.httpPort

client/sniffer/src/main/java/org/elasticsearch/client/sniff/Sniffer.java

+37-1
Original file line numberDiff line numberDiff line change
@@ -27,12 +27,16 @@
2727

2828
import java.io.Closeable;
2929
import java.io.IOException;
30+
import java.security.AccessController;
31+
import java.security.PrivilegedAction;
3032
import java.util.List;
3133
import java.util.concurrent.Executors;
3234
import java.util.concurrent.ScheduledExecutorService;
3335
import java.util.concurrent.ScheduledFuture;
36+
import java.util.concurrent.ThreadFactory;
3437
import java.util.concurrent.TimeUnit;
3538
import java.util.concurrent.atomic.AtomicBoolean;
39+
import java.util.concurrent.atomic.AtomicInteger;
3640

3741
/**
3842
* Class responsible for sniffing nodes from some source (default is elasticsearch itself) and setting them to a provided instance of
@@ -45,6 +49,7 @@
4549
public class Sniffer implements Closeable {
4650

4751
private static final Log logger = LogFactory.getLog(Sniffer.class);
52+
private static final String SNIFFER_THREAD_NAME = "es_rest_client_sniffer";
4853

4954
private final Task task;
5055

@@ -79,7 +84,8 @@ private Task(HostsSniffer hostsSniffer, RestClient restClient, long sniffInterva
7984
this.restClient = restClient;
8085
this.sniffIntervalMillis = sniffIntervalMillis;
8186
this.sniffAfterFailureDelayMillis = sniffAfterFailureDelayMillis;
82-
this.scheduledExecutorService = Executors.newScheduledThreadPool(1);
87+
SnifferThreadFactory threadFactory = new SnifferThreadFactory(SNIFFER_THREAD_NAME);
88+
this.scheduledExecutorService = Executors.newScheduledThreadPool(1, threadFactory);
8389
scheduleNextRun(0);
8490
}
8591

@@ -151,4 +157,34 @@ synchronized void shutdown() {
151157
public static SnifferBuilder builder(RestClient restClient) {
152158
return new SnifferBuilder(restClient);
153159
}
160+
161+
private static class SnifferThreadFactory implements ThreadFactory {
162+
163+
private final AtomicInteger threadNumber = new AtomicInteger(1);
164+
private final String namePrefix;
165+
private final ThreadFactory originalThreadFactory;
166+
167+
private SnifferThreadFactory(String namePrefix) {
168+
this.namePrefix = namePrefix;
169+
this.originalThreadFactory = AccessController.doPrivileged(new PrivilegedAction<ThreadFactory>() {
170+
@Override
171+
public ThreadFactory run() {
172+
return Executors.defaultThreadFactory();
173+
}
174+
});
175+
}
176+
177+
@Override
178+
public Thread newThread(final Runnable r) {
179+
return AccessController.doPrivileged(new PrivilegedAction<Thread>() {
180+
@Override
181+
public Thread run() {
182+
Thread t = originalThreadFactory.newThread(r);
183+
t.setName(namePrefix + "[T#" + threadNumber.getAndIncrement() + "]");
184+
t.setDaemon(true);
185+
return t;
186+
}
187+
});
188+
}
189+
}
154190
}

core/src/main/java/org/elasticsearch/Version.java

+5-1
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,8 @@ public class Version implements Comparable<Version> {
9898
public static final Version V_5_6_2 = new Version(V_5_6_2_ID, org.apache.lucene.util.Version.LUCENE_6_6_1);
9999
public static final int V_5_6_3_ID = 5060399;
100100
public static final Version V_5_6_3 = new Version(V_5_6_3_ID, org.apache.lucene.util.Version.LUCENE_6_6_1);
101+
public static final int V_5_6_4_ID = 5060499;
102+
public static final Version V_5_6_4 = new Version(V_5_6_4_ID, org.apache.lucene.util.Version.LUCENE_6_6_1);
101103
public static final int V_6_0_0_alpha1_ID = 6000001;
102104
public static final Version V_6_0_0_alpha1 = new Version(V_6_0_0_alpha1_ID, org.apache.lucene.util.Version.LUCENE_7_0_0);
103105
public static final int V_6_0_0_alpha2_ID = 6000002;
@@ -111,7 +113,7 @@ public class Version implements Comparable<Version> {
111113
new Version(V_6_0_0_rc1_ID, org.apache.lucene.util.Version.LUCENE_7_0_0);
112114
public static final int V_6_0_0_rc2_ID = 6000052;
113115
public static final Version V_6_0_0_rc2 =
114-
new Version(V_6_0_0_rc2_ID, org.apache.lucene.util.Version.LUCENE_7_0_0);
116+
new Version(V_6_0_0_rc2_ID, org.apache.lucene.util.Version.LUCENE_7_0_1);
115117
public static final int V_6_1_0_ID = 6010099;
116118
public static final Version V_6_1_0 = new Version(V_6_1_0_ID, org.apache.lucene.util.Version.LUCENE_7_1_0);
117119
public static final Version CURRENT = V_6_1_0;
@@ -141,6 +143,8 @@ public static Version fromId(int id) {
141143
return V_6_0_0_alpha2;
142144
case V_6_0_0_alpha1_ID:
143145
return V_6_0_0_alpha1;
146+
case V_5_6_4_ID:
147+
return V_5_6_4;
144148
case V_5_6_3_ID:
145149
return V_5_6_3;
146150
case V_5_6_2_ID:

core/src/main/java/org/elasticsearch/cluster/routing/OperationRouting.java

+8
Original file line numberDiff line numberDiff line change
@@ -199,12 +199,20 @@ private ShardIterator preferenceActiveShardIterator(IndexShardRoutingTable index
199199
case LOCAL:
200200
return indexShard.preferNodeActiveInitializingShardsIt(Collections.singleton(localNodeId));
201201
case PRIMARY:
202+
deprecationLogger.deprecated("[_primary] has been deprecated in 6.1+, and will be removed in 7.0; " +
203+
"use [_only_nodes] or [_prefer_nodes]");
202204
return indexShard.primaryActiveInitializingShardIt();
203205
case REPLICA:
206+
deprecationLogger.deprecated("[_replica] has been deprecated in 6.1+, and will be removed in 7.0; " +
207+
"use [_only_nodes] or [_prefer_nodes]");
204208
return indexShard.replicaActiveInitializingShardIt();
205209
case PRIMARY_FIRST:
210+
deprecationLogger.deprecated("[_primary_first] has been deprecated in 6.1+, and will be removed in 7.0; " +
211+
"use [_only_nodes] or [_prefer_nodes]");
206212
return indexShard.primaryFirstActiveInitializingShardsIt();
207213
case REPLICA_FIRST:
214+
deprecationLogger.deprecated("[_replica_first] has been deprecated in 6.1+, and will be removed in 7.0; " +
215+
"use [_only_nodes] or [_prefer_nodes]");
208216
return indexShard.replicaFirstActiveInitializingShardsIt();
209217
case ONLY_LOCAL:
210218
return indexShard.onlyNodeActiveInitializingShardsIt(localNodeId);

core/src/main/java/org/elasticsearch/cluster/routing/UnassignedInfo.java

+14-2
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,8 @@
3030
import org.elasticsearch.common.io.stream.Writeable;
3131
import org.elasticsearch.common.joda.FormatDateTimeFormatter;
3232
import org.elasticsearch.common.joda.Joda;
33+
import org.elasticsearch.common.logging.DeprecationLogger;
34+
import org.elasticsearch.common.logging.Loggers;
3335
import org.elasticsearch.common.settings.Setting;
3436
import org.elasticsearch.common.settings.Setting.Property;
3537
import org.elasticsearch.common.settings.Settings;
@@ -47,11 +49,21 @@
4749
*/
4850
public final class UnassignedInfo implements ToXContentFragment, Writeable {
4951

52+
private static final DeprecationLogger DEPRECATION_LOGGER = new DeprecationLogger(Loggers.getLogger(UnassignedInfo.class));
53+
5054
public static final FormatDateTimeFormatter DATE_TIME_FORMATTER = Joda.forPattern("dateOptionalTime");
5155

5256
public static final Setting<TimeValue> INDEX_DELAYED_NODE_LEFT_TIMEOUT_SETTING =
53-
Setting.timeSetting("index.unassigned.node_left.delayed_timeout", TimeValue.timeValueMinutes(1), Property.Dynamic,
54-
Property.IndexScope);
57+
new Setting<>("index.unassigned.node_left.delayed_timeout", (s) -> TimeValue.timeValueMinutes(1).getStringRep(), (s) -> {
58+
TimeValue parsedValue = TimeValue.parseTimeValue(s, "index.unassigned.node_left.delayed_timeout");
59+
if (parsedValue.getNanos() < 0) {
60+
DEPRECATION_LOGGER.deprecated(
61+
"Negative values for index.unassigned.node_left.delayed_timeout [{}]" +
62+
" are deprecated and should now be set to \"0\".", s);
63+
}
64+
return parsedValue;
65+
}, Property.Dynamic, Property.IndexScope);
66+
5567
/**
5668
* Reason why the shard is in unassigned state.
5769
* <p>

core/src/main/java/org/elasticsearch/cluster/routing/allocation/decider/AwarenessAllocationDecider.java

+6-5
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
package org.elasticsearch.cluster.routing.allocation.decider;
2121

2222
import java.util.HashMap;
23+
import java.util.List;
2324
import java.util.Map;
2425

2526
import com.carrotsearch.hppc.ObjectIntHashMap;
@@ -85,7 +86,7 @@ public class AwarenessAllocationDecider extends AllocationDecider {
8586

8687
private volatile String[] awarenessAttributes;
8788

88-
private volatile Map<String, String[]> forcedAwarenessAttributes;
89+
private volatile Map<String, List<String>> forcedAwarenessAttributes;
8990

9091
public AwarenessAllocationDecider(Settings settings, ClusterSettings clusterSettings) {
9192
super(settings);
@@ -97,11 +98,11 @@ public AwarenessAllocationDecider(Settings settings, ClusterSettings clusterSett
9798
}
9899

99100
private void setForcedAwarenessAttributes(Settings forceSettings) {
100-
Map<String, String[]> forcedAwarenessAttributes = new HashMap<>();
101+
Map<String, List<String>> forcedAwarenessAttributes = new HashMap<>();
101102
Map<String, Settings> forceGroups = forceSettings.getAsGroups();
102103
for (Map.Entry<String, Settings> entry : forceGroups.entrySet()) {
103-
String[] aValues = entry.getValue().getAsArray("values");
104-
if (aValues.length > 0) {
104+
List<String> aValues = entry.getValue().getAsList("values");
105+
if (aValues.size() > 0) {
105106
forcedAwarenessAttributes.put(entry.getKey(), aValues);
106107
}
107108
}
@@ -169,7 +170,7 @@ private Decision underCapacity(ShardRouting shardRouting, RoutingNode node, Rout
169170
}
170171

171172
int numberOfAttributes = nodesPerAttribute.size();
172-
String[] fullValues = forcedAwarenessAttributes.get(awarenessAttribute);
173+
List<String> fullValues = forcedAwarenessAttributes.get(awarenessAttribute);
173174
if (fullValues != null) {
174175
for (String fullValue : fullValues) {
175176
if (!shardPerAttribute.containsKey(fullValue)) {

core/src/main/java/org/elasticsearch/common/settings/Setting.java

+7-7
Original file line numberDiff line numberDiff line change
@@ -804,14 +804,14 @@ private static class ListSetting<T> extends Setting<List<T>> {
804804

805805
private ListSetting(String key, Function<Settings, List<String>> defaultStringValue, Function<String, List<T>> parser,
806806
Property... properties) {
807-
super(new ListKey(key), (s) -> Setting.arrayToParsableString(defaultStringValue.apply(s).toArray(Strings.EMPTY_ARRAY)), parser,
807+
super(new ListKey(key), (s) -> Setting.arrayToParsableString(defaultStringValue.apply(s)), parser,
808808
properties);
809809
this.defaultStringValue = defaultStringValue;
810810
}
811811

812812
@Override
813813
public String getRaw(Settings settings) {
814-
String[] array = settings.getAsArray(getKey(), null);
814+
List<String> array = settings.getAsList(getKey(), null);
815815
return array == null ? defaultValue.apply(settings) : arrayToParsableString(array);
816816
}
817817

@@ -823,11 +823,11 @@ boolean hasComplexMatcher() {
823823
@Override
824824
public void diff(Settings.Builder builder, Settings source, Settings defaultSettings) {
825825
if (exists(source) == false) {
826-
String[] asArray = defaultSettings.getAsArray(getKey(), null);
827-
if (asArray == null) {
828-
builder.putArray(getKey(), defaultStringValue.apply(defaultSettings));
826+
List<String> asList = defaultSettings.getAsList(getKey(), null);
827+
if (asList == null) {
828+
builder.putList(getKey(), defaultStringValue.apply(defaultSettings));
829829
} else {
830-
builder.putArray(getKey(), asArray);
830+
builder.putList(getKey(), asList);
831831
}
832832
}
833833
}
@@ -1087,7 +1087,7 @@ private static List<String> parseableStringToList(String parsableString) {
10871087
}
10881088
}
10891089

1090-
private static String arrayToParsableString(String[] array) {
1090+
private static String arrayToParsableString(List<String> array) {
10911091
try {
10921092
XContentBuilder builder = XContentBuilder.builder(XContentType.JSON.xContent());
10931093
builder.startArray();

0 commit comments

Comments
 (0)