Skip to content

Commit fe756a3

Browse files
jasontedorjasontedor
committed
Merge branch 'master' into unknown-or-invalid-settings-updates
* master: [TEST] AwaitsFix QueryRescorerIT.testRescoreAfterCollapse Decouple XContentType from StreamInput/Output (elastic#28927) Remove BytesRef usage from XContentParser and its subclasses (elastic#28792) [DOCS] Correct typo in configuration (elastic#28903) Fix incorrect datemath example (elastic#28904) Add a usage example of the JLH score (elastic#28905) Wrap stream passed to createParser in try-with-resources (elastic#28897) Rescore collapsed documents (elastic#28521) Fix (simple)_query_string to ignore removed terms (elastic#28871) [Docs] Fix typo in composite aggregation (elastic#28891) Try if tombstone is eligable for pruning before locking on it's key (elastic#28767)
2 parents 9e55fac + 9d4f09d commit fe756a3

File tree

56 files changed

+421
-247
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

56 files changed

+421
-247
lines changed

‎docs/reference/aggregations/bucket/composite-aggregation.asciidoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -352,7 +352,7 @@ and in ascending order when comparing values from the `terms` source.
352352

353353
The `size` parameter can be set to define how many composite buckets should be returned.
354354
Each composite bucket is considered as a single bucket so setting a size of 10 will return the
355-
first 1O composite buckets created from the values source.
355+
first 10 composite buckets created from the values source.
356356
The response contains the values for each composite bucket in an array containing the values extracted
357357
from each value source.
358358

‎docs/reference/aggregations/bucket/significantterms-aggregation.asciidoc

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -327,6 +327,15 @@ However, the `size` and `shard size` settings covered in the next section provid
327327
==== Parameters
328328

329329
===== JLH score
330+
The JLH score can be used as a significance score by adding the parameter
331+
332+
[source,js]
333+
--------------------------------------------------
334+
335+
"jlh": {
336+
}
337+
--------------------------------------------------
338+
// NOTCONSOLE
330339

331340
The scores are derived from the doc frequencies in _foreground_ and _background_ sets. The _absolute_ change in popularity (foregroundPercent - backgroundPercent) would favor common terms whereas the _relative_ change in popularity (foregroundPercent/ backgroundPercent) would favor rare terms. Rare vs common is essentially a precision vs recall balance and so the absolute and relative changes are multiplied to provide a sweet spot between precision and recall.
332341

‎docs/reference/api-conventions.asciidoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -220,7 +220,7 @@ Assuming `now` is `2001-01-01 12:00:00`, some examples are:
220220
`now+1h`:: `now` in milliseconds plus one hour. Resolves to: `2001-01-01 13:00:00`
221221
`now-1h`:: `now` in milliseconds minus one hour. Resolves to: `2001-01-01 11:00:00`
222222
`now-1h/d`:: `now` in milliseconds minus one hour, rounded down to UTC 00:00. Resolves to: `2001-01-01 00:00:00``
223-
`2001-01-01\|\|+1M/d`:: `now` in milliseconds plus one month. Resolves to: `2001-02-01 00:00:00`
223+
`2001.02.01\|\|+1M/d`:: `2001-02-01` in milliseconds plus one month. Resolves to: `2001-03-01 00:00:00`
224224

225225
[float]
226226
[[common-options-response-filtering]]

‎docs/reference/setup/configuration.asciidoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ command line or via your shell profile.
3838
For the package distributions, the config directory location defaults to
3939
`/etc/elasticsearch`. The location of the config directory can also be changed
4040
via the `ES_PATH_CONF` environment variable, but note that setting this in your
41-
shell is not sufficient. Instead, this variabled is sourced from
41+
shell is not sufficient. Instead, this variable is sourced from
4242
`/etc/default/elasticsearch` (for the Debian package) and
4343
`/etc/sysconfig/elasticsearch` (for the RPM package). You will need to edit the
4444
`ES_PATH_CONF=/etc/elasticsearch` entry in one of these files accordingly to

‎modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/JsonProcessor.java

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@
3838
import org.elasticsearch.ingest.Processor;
3939

4040
import java.io.IOException;
41+
import java.io.InputStream;
4142
import java.util.Map;
4243

4344
import static org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException;
@@ -77,8 +78,9 @@ boolean isAddToRoot() {
7778
public void execute(IngestDocument document) throws Exception {
7879
Object fieldValue = document.getFieldValue(field, Object.class);
7980
BytesReference bytesRef = (fieldValue == null) ? new BytesArray("null") : new BytesArray(fieldValue.toString());
80-
try (XContentParser parser = JsonXContent.jsonXContent
81-
.createParser(NamedXContentRegistry.EMPTY, DeprecationHandler.THROW_UNSUPPORTED_OPERATION, bytesRef.streamInput())) {
81+
try (InputStream stream = bytesRef.streamInput();
82+
XContentParser parser = JsonXContent.jsonXContent
83+
.createParser(NamedXContentRegistry.EMPTY, DeprecationHandler.THROW_UNSUPPORTED_OPERATION, stream)) {
8284
XContentParser.Token token = parser.nextToken();
8385
Object value = null;
8486
if (token == XContentParser.Token.VALUE_NULL) {

‎modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/ScriptProcessor.java

Lines changed: 17 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@
3636
import org.elasticsearch.script.ScriptException;
3737
import org.elasticsearch.script.ScriptService;
3838

39+
import java.io.InputStream;
3940
import java.util.Arrays;
4041
import java.util.Map;
4142

@@ -97,21 +98,23 @@ public Factory(ScriptService scriptService) {
9798
@Override
9899
public ScriptProcessor create(Map<String, Processor.Factory> registry, String processorTag,
99100
Map<String, Object> config) throws Exception {
100-
XContentBuilder builder = XContentBuilder.builder(JsonXContent.jsonXContent).map(config);
101-
XContentParser parser = XContentType.JSON.xContent().createParser(NamedXContentRegistry.EMPTY,
102-
LoggingDeprecationHandler.INSTANCE, builder.bytes().streamInput());
103-
Script script = Script.parse(parser);
104-
105-
Arrays.asList("id", "source", "inline", "lang", "params", "options").forEach(config::remove);
106-
107-
// verify script is able to be compiled before successfully creating processor.
108-
try {
109-
scriptService.compile(script, ExecutableScript.INGEST_CONTEXT);
110-
} catch (ScriptException e) {
111-
throw newConfigurationException(TYPE, processorTag, null, e);
101+
try (XContentBuilder builder = XContentBuilder.builder(JsonXContent.jsonXContent).map(config);
102+
InputStream stream = builder.bytes().streamInput();
103+
XContentParser parser = XContentType.JSON.xContent().createParser(NamedXContentRegistry.EMPTY,
104+
LoggingDeprecationHandler.INSTANCE, stream)) {
105+
Script script = Script.parse(parser);
106+
107+
Arrays.asList("id", "source", "inline", "lang", "params", "options").forEach(config::remove);
108+
109+
// verify script is able to be compiled before successfully creating processor.
110+
try {
111+
scriptService.compile(script, ExecutableScript.INGEST_CONTEXT);
112+
} catch (ScriptException e) {
113+
throw newConfigurationException(TYPE, processorTag, null, e);
114+
}
115+
116+
return new ScriptProcessor(processorTag, script, scriptService);
112117
}
113-
114-
return new ScriptProcessor(processorTag, script, scriptService);
115118
}
116119
}
117120
}

‎modules/percolator/src/main/java/org/elasticsearch/percolator/PercolateQueryBuilder.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -274,7 +274,7 @@ public PercolateQueryBuilder(String field, String documentType, String indexedDo
274274
}
275275
if (documents.isEmpty() == false) {
276276
if (in.getVersion().onOrAfter(Version.V_5_3_0)) {
277-
documentXContentType = XContentType.readFrom(in);
277+
documentXContentType = in.readEnum(XContentType.class);
278278
} else {
279279
documentXContentType = XContentFactory.xContentType(documents.iterator().next());
280280
}
@@ -331,7 +331,7 @@ protected void doWriteTo(StreamOutput out) throws IOException {
331331
out.writeOptionalBytesReference(doc);
332332
}
333333
if (documents.isEmpty() == false && out.getVersion().onOrAfter(Version.V_5_3_0)) {
334-
documentXContentType.writeTo(out);
334+
out.writeEnum(documentXContentType);
335335
}
336336
}
337337

‎modules/reindex/src/main/java/org/elasticsearch/index/reindex/RestReindexAction.java

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@
4141
import org.elasticsearch.script.Script;
4242

4343
import java.io.IOException;
44+
import java.io.InputStream;
4445
import java.util.List;
4546
import java.util.Map;
4647
import java.util.regex.Matcher;
@@ -74,8 +75,9 @@ public class RestReindexAction extends AbstractBaseReindexRestHandler<ReindexReq
7475
request.setRemoteInfo(buildRemoteInfo(source));
7576
XContentBuilder builder = XContentFactory.contentBuilder(parser.contentType());
7677
builder.map(source);
77-
try (XContentParser innerParser = parser.contentType().xContent()
78-
.createParser(parser.getXContentRegistry(), parser.getDeprecationHandler(), builder.bytes().streamInput())) {
78+
try (InputStream stream = builder.bytes().streamInput();
79+
XContentParser innerParser = parser.contentType().xContent()
80+
.createParser(parser.getXContentRegistry(), parser.getDeprecationHandler(), stream)) {
7981
request.getSearchRequest().source().parseXContent(innerParser);
8082
}
8183
};

‎modules/reindex/src/main/java/org/elasticsearch/index/reindex/TransportReindexAction.java

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@
7272
import org.elasticsearch.transport.TransportService;
7373

7474
import java.io.IOException;
75+
import java.io.InputStream;
7576
import java.io.UncheckedIOException;
7677
import java.util.ArrayList;
7778
import java.util.List;
@@ -338,9 +339,9 @@ protected RequestWrapper<IndexRequest> buildRequest(ScrollableHitSource.Hit doc)
338339
final XContentType mainRequestXContentType = mainRequest.getDestination().getContentType();
339340
if (mainRequestXContentType != null && doc.getXContentType() != mainRequestXContentType) {
340341
// we need to convert
341-
try (XContentParser parser = sourceXContentType.xContent()
342-
.createParser(NamedXContentRegistry.EMPTY,
343-
DeprecationHandler.THROW_UNSUPPORTED_OPERATION, doc.getSource().streamInput());
342+
try (InputStream stream = doc.getSource().streamInput();
343+
XContentParser parser = sourceXContentType.xContent()
344+
.createParser(NamedXContentRegistry.EMPTY, DeprecationHandler.THROW_UNSUPPORTED_OPERATION, stream);
344345
XContentBuilder builder = XContentBuilder.builder(mainRequestXContentType.xContent())) {
345346
parser.nextToken();
346347
builder.copyCurrentStructure(parser);

‎rest-api-spec/src/main/resources/rest-api-spec/test/search/110_field_collapsing.yml

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -237,28 +237,6 @@ setup:
237237
search_after: [6]
238238
sort: [{ sort: desc }]
239239

240-
---
241-
"field collapsing and rescore":
242-
243-
- skip:
244-
version: " - 5.2.99"
245-
reason: this uses a new API that has been added in 5.3
246-
247-
- do:
248-
catch: /cannot use \`collapse\` in conjunction with \`rescore\`/
249-
search:
250-
index: test
251-
type: test
252-
body:
253-
collapse: { field: numeric_group }
254-
rescore:
255-
window_size: 20
256-
query:
257-
rescore_query:
258-
match_all: {}
259-
query_weight: 1
260-
rescore_query_weight: 2
261-
262240
---
263241
"no hits and inner_hits":
264242

‎server/src/main/java/org/elasticsearch/action/admin/cluster/storedscripts/PutStoredScriptRequest.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,7 @@ public void readFrom(StreamInput in) throws IOException {
123123
id = in.readOptionalString();
124124
content = in.readBytesReference();
125125
if (in.getVersion().onOrAfter(Version.V_5_3_0)) {
126-
xContentType = XContentType.readFrom(in);
126+
xContentType = in.readEnum(XContentType.class);
127127
} else {
128128
xContentType = XContentFactory.xContentType(content);
129129
}
@@ -145,7 +145,7 @@ public void writeTo(StreamOutput out) throws IOException {
145145
out.writeOptionalString(id);
146146
out.writeBytesReference(content);
147147
if (out.getVersion().onOrAfter(Version.V_5_3_0)) {
148-
xContentType.writeTo(out);
148+
out.writeEnum(xContentType);
149149
}
150150
if (out.getVersion().onOrAfter(Version.V_6_0_0_alpha2)) {
151151
out.writeOptionalString(context);

‎server/src/main/java/org/elasticsearch/action/admin/indices/mapping/put/PutMappingRequest.java

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
import org.elasticsearch.index.Index;
4141

4242
import java.io.IOException;
43+
import java.io.InputStream;
4344
import java.io.UncheckedIOException;
4445
import java.util.Arrays;
4546
import java.util.Map;
@@ -323,7 +324,9 @@ public void writeTo(StreamOutput out) throws IOException {
323324
@Override
324325
public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
325326
if (source != null) {
326-
builder.rawValue(new BytesArray(source).streamInput(), XContentType.JSON);
327+
try (InputStream stream = new BytesArray(source).streamInput()) {
328+
builder.rawValue(stream, XContentType.JSON);
329+
}
327330
} else {
328331
builder.startObject().endObject();
329332
}

‎server/src/main/java/org/elasticsearch/action/bulk/BulkRequest.java

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@
4949
import org.elasticsearch.search.fetch.subphase.FetchSourceContext;
5050

5151
import java.io.IOException;
52+
import java.io.InputStream;
5253
import java.util.ArrayList;
5354
import java.util.HashSet;
5455
import java.util.List;
@@ -305,9 +306,9 @@ public BulkRequest add(BytesReference data, @Nullable String defaultIndex, @Null
305306

306307
// now parse the action
307308
// EMPTY is safe here because we never call namedObject
308-
try (XContentParser parser = xContent
309-
.createParser(NamedXContentRegistry.EMPTY, LoggingDeprecationHandler.INSTANCE,
310-
data.slice(from, nextMarker - from).streamInput())) {
309+
try (InputStream stream = data.slice(from, nextMarker - from).streamInput();
310+
XContentParser parser = xContent
311+
.createParser(NamedXContentRegistry.EMPTY, LoggingDeprecationHandler.INSTANCE, stream)) {
311312
// move pointers
312313
from = nextMarker + 1;
313314

@@ -431,8 +432,9 @@ public BulkRequest add(BytesReference data, @Nullable String defaultIndex, @Null
431432
.routing(routing)
432433
.parent(parent);
433434
// EMPTY is safe here because we never call namedObject
434-
try (XContentParser sliceParser = xContent.createParser(NamedXContentRegistry.EMPTY,
435-
LoggingDeprecationHandler.INSTANCE, sliceTrimmingCarriageReturn(data, from, nextMarker, xContentType).streamInput())) {
435+
try (InputStream dataStream = sliceTrimmingCarriageReturn(data, from, nextMarker, xContentType).streamInput();
436+
XContentParser sliceParser = xContent.createParser(NamedXContentRegistry.EMPTY,
437+
LoggingDeprecationHandler.INSTANCE, dataStream)) {
436438
updateRequest.fromXContent(sliceParser);
437439
}
438440
if (fetchSourceContext != null) {

‎server/src/main/java/org/elasticsearch/action/index/IndexRequest.java

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -542,7 +542,11 @@ public void readFrom(StreamInput in) throws IOException {
542542
pipeline = in.readOptionalString();
543543
isRetry = in.readBoolean();
544544
autoGeneratedTimestamp = in.readLong();
545-
contentType = in.readOptionalWriteable(XContentType::readFrom);
545+
if (in.readBoolean()) {
546+
contentType = in.readEnum(XContentType.class);
547+
} else {
548+
contentType = null;
549+
}
546550
}
547551

548552
@Override
@@ -566,7 +570,12 @@ public void writeTo(StreamOutput out) throws IOException {
566570
out.writeOptionalString(pipeline);
567571
out.writeBoolean(isRetry);
568572
out.writeLong(autoGeneratedTimestamp);
569-
out.writeOptionalWriteable(contentType);
573+
if (contentType != null) {
574+
out.writeBoolean(true);
575+
out.writeEnum(contentType);
576+
} else {
577+
out.writeBoolean(false);
578+
}
570579
}
571580

572581
@Override

‎server/src/main/java/org/elasticsearch/action/ingest/PutPipelineRequest.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ public void readFrom(StreamInput in) throws IOException {
8181
id = in.readString();
8282
source = in.readBytesReference();
8383
if (in.getVersion().onOrAfter(Version.V_5_3_0)) {
84-
xContentType = XContentType.readFrom(in);
84+
xContentType = in.readEnum(XContentType.class);
8585
} else {
8686
xContentType = XContentFactory.xContentType(source);
8787
}
@@ -93,7 +93,7 @@ public void writeTo(StreamOutput out) throws IOException {
9393
out.writeString(id);
9494
out.writeBytesReference(source);
9595
if (out.getVersion().onOrAfter(Version.V_5_3_0)) {
96-
xContentType.writeTo(out);
96+
out.writeEnum(xContentType);
9797
}
9898
}
9999
}

‎server/src/main/java/org/elasticsearch/action/ingest/SimulatePipelineRequest.java

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ public SimulatePipelineRequest(BytesReference source, XContentType xContentType)
7676
verbose = in.readBoolean();
7777
source = in.readBytesReference();
7878
if (in.getVersion().onOrAfter(Version.V_5_3_0)) {
79-
xContentType = XContentType.readFrom(in);
79+
xContentType = in.readEnum(XContentType.class);
8080
} else {
8181
xContentType = XContentFactory.xContentType(source);
8282
}
@@ -123,7 +123,7 @@ public void writeTo(StreamOutput out) throws IOException {
123123
out.writeBoolean(verbose);
124124
out.writeBytesReference(source);
125125
if (out.getVersion().onOrAfter(Version.V_5_3_0)) {
126-
xContentType.writeTo(out);
126+
out.writeEnum(xContentType);
127127
}
128128
}
129129

‎server/src/main/java/org/elasticsearch/action/search/MultiSearchRequest.java

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@
3636

3737
import java.io.ByteArrayOutputStream;
3838
import java.io.IOException;
39+
import java.io.InputStream;
3940
import java.util.ArrayList;
4041
import java.util.List;
4142
import java.util.Locale;
@@ -207,9 +208,8 @@ public static void readMultiLineFormat(BytesReference data,
207208
IndicesOptions defaultOptions = SearchRequest.DEFAULT_INDICES_OPTIONS;
208209
// now parse the action
209210
if (nextMarker - from > 0) {
210-
try (XContentParser parser = xContent
211-
.createParser(registry, LoggingDeprecationHandler.INSTANCE,
212-
data.slice(from, nextMarker - from).streamInput())) {
211+
try (InputStream stream = data.slice(from, nextMarker - from).streamInput();
212+
XContentParser parser = xContent.createParser(registry, LoggingDeprecationHandler.INSTANCE, stream)) {
213213
Map<String, Object> source = parser.map();
214214
for (Map.Entry<String, Object> entry : source.entrySet()) {
215215
Object value = entry.getValue();
@@ -245,7 +245,8 @@ public static void readMultiLineFormat(BytesReference data,
245245
break;
246246
}
247247
BytesReference bytes = data.slice(from, nextMarker - from);
248-
try (XContentParser parser = xContent.createParser(registry, LoggingDeprecationHandler.INSTANCE, bytes.streamInput())) {
248+
try (InputStream stream = bytes.streamInput();
249+
XContentParser parser = xContent.createParser(registry, LoggingDeprecationHandler.INSTANCE, stream)) {
249250
consumer.accept(searchRequest, parser);
250251
}
251252
// move pointers

‎server/src/main/java/org/elasticsearch/action/termvectors/TermVectorsFields.java

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -130,21 +130,21 @@ public final class TermVectorsFields extends Fields {
130130
* @param termVectors Stores the actual term vectors as a {@link BytesRef}.
131131
*/
132132
public TermVectorsFields(BytesReference headerRef, BytesReference termVectors) throws IOException {
133-
StreamInput header = headerRef.streamInput();
134-
fieldMap = new ObjectLongHashMap<>();
135-
// here we read the header to fill the field offset map
136-
String headerString = header.readString();
137-
assert headerString.equals("TV");
138-
int version = header.readInt();
139-
assert version == -1;
140-
hasTermStatistic = header.readBoolean();
141-
hasFieldStatistic = header.readBoolean();
142-
hasScores = header.readBoolean();
143-
final int numFields = header.readVInt();
144-
for (int i = 0; i < numFields; i++) {
145-
fieldMap.put((header.readString()), header.readVLong());
133+
try (StreamInput header = headerRef.streamInput()) {
134+
fieldMap = new ObjectLongHashMap<>();
135+
// here we read the header to fill the field offset map
136+
String headerString = header.readString();
137+
assert headerString.equals("TV");
138+
int version = header.readInt();
139+
assert version == -1;
140+
hasTermStatistic = header.readBoolean();
141+
hasFieldStatistic = header.readBoolean();
142+
hasScores = header.readBoolean();
143+
final int numFields = header.readVInt();
144+
for (int i = 0; i < numFields; i++) {
145+
fieldMap.put((header.readString()), header.readVLong());
146+
}
146147
}
147-
header.close();
148148
// reference to the term vector data
149149
this.termVectors = termVectors;
150150
}

0 commit comments

Comments
 (0)