Skip to content

Commit 5d60e36

Browse files
committed
Merge branch 'master' into align-thread-pool-info-cat-api
* master: Reindex: Fix error in delete-by-query rest spec (elastic#29318) Improve similarity integration. (elastic#29187) Fix some query extraction bugs. (elastic#29283) [Docs] Correct experimental note formatting Move Nullable into core (elastic#29341) [Docs] Update getting-started.asciidoc (elastic#29294) Elasticsearch 6.3.0 is now on Lucene 7.3. [DOCS] Refer back to index API for full-document updates in _update API section (elastic#28677) Fix missing comma in ingest-node.asciidoc (elastic#29343) Improve exception handling on TransportMasterNodeAction (elastic#29314) Don't break allocation if resize source index is missing (elastic#29311) Use fixture to test repository-s3 plugin (elastic#29296) Fix NDCG for empty search results (elastic#29267) Pass through script params in scripted metric agg (elastic#29154) Fix Eclipse build. Upgrade to lucene-7.3.0-snapshot-98a6b3d. (elastic#29298) Painless: Remove extraneous INLINE constant. (elastic#29340)
2 parents 1112584 + 4db6fc9 commit 5d60e36

File tree

166 files changed

+2134
-1366
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

166 files changed

+2134
-1366
lines changed

buildSrc/src/main/groovy/org/elasticsearch/gradle/plugin/PluginBuildPlugin.groovy

+1-1
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@ public class PluginBuildPlugin extends BuildPlugin {
9595
// we "upgrade" these optional deps to provided for plugins, since they will run
9696
// with a full elasticsearch server that includes optional deps
9797
compileOnly "org.locationtech.spatial4j:spatial4j:${project.versions.spatial4j}"
98-
compileOnly "com.vividsolutions:jts:${project.versions.jts}"
98+
compileOnly "org.locationtech.jts:jts-core:${project.versions.jts}"
9999
compileOnly "org.apache.logging.log4j:log4j-api:${project.versions.log4j}"
100100
compileOnly "org.apache.logging.log4j:log4j-core:${project.versions.log4j}"
101101
compileOnly "org.elasticsearch:jna:${project.versions.jna}"

buildSrc/version.properties

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
elasticsearch = 7.0.0-alpha1
2-
lucene = 7.2.1
2+
lucene = 7.3.0-snapshot-98a6b3d
33

44
# optional dependencies
5-
spatial4j = 0.6
6-
jts = 1.13
5+
spatial4j = 0.7
6+
jts = 1.15.0
77
jackson = 2.8.10
88
snakeyaml = 1.17
99
# when updating log4j, please update also docs/java-api/index.asciidoc

docs/Versions.asciidoc

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
:version: 7.0.0-alpha1
22
:major-version: 7.x
3-
:lucene_version: 7.2.1
4-
:lucene_version_path: 7_2_1
3+
:lucene_version: 7.3.0
4+
:lucene_version_path: 7_3_0
55
:branch: master
66
:jdk: 1.8.0_131
77
:jdk_major: 8

docs/java-api/query-dsl/geo-shape-query.asciidoc

+5-5
Original file line numberDiff line numberDiff line change
@@ -12,13 +12,13 @@ to your classpath in order to use this type:
1212
<dependency>
1313
<groupId>org.locationtech.spatial4j</groupId>
1414
<artifactId>spatial4j</artifactId>
15-
<version>0.6</version> <1>
15+
<version>0.7</version> <1>
1616
</dependency>
1717
1818
<dependency>
19-
<groupId>com.vividsolutions</groupId>
20-
<artifactId>jts</artifactId>
21-
<version>1.13</version> <2>
19+
<groupId>org.locationtech.jts</groupId>
20+
<artifactId>jts-core</artifactId>
21+
<version>1.15.0</version> <2>
2222
<exclusions>
2323
<exclusion>
2424
<groupId>xerces</groupId>
@@ -28,7 +28,7 @@ to your classpath in order to use this type:
2828
</dependency>
2929
-----------------------------------------------
3030
<1> check for updates in http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.locationtech.spatial4j%22%20AND%20a%3A%22spatial4j%22[Maven Central]
31-
<2> check for updates in http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.vividsolutions%22%20AND%20a%3A%22jts%22[Maven Central]
31+
<2> check for updates in http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.locationtech.jts%22%20AND%20a%3A%22jts-core%22[Maven Central]
3232

3333
[source,java]
3434
--------------------------------------------------

docs/reference/docs/update.asciidoc

+5-2
Original file line numberDiff line numberDiff line change
@@ -118,8 +118,11 @@ POST test/_doc/1/_update
118118

119119
The update API also support passing a partial document,
120120
which will be merged into the existing document (simple recursive merge,
121-
inner merging of objects, replacing core "keys/values" and arrays). For
122-
example:
121+
inner merging of objects, replacing core "keys/values" and arrays).
122+
To fully replace the existing document, the <<docs-index_,`index` API>> should
123+
be used instead.
124+
The following partial update adds a new field to the
125+
existing document:
123126

124127
[source,js]
125128
--------------------------------------------------

docs/reference/getting-started.asciidoc

+1-1
Original file line numberDiff line numberDiff line change
@@ -777,7 +777,7 @@ GET /bank/_search
777777
// CONSOLE
778778
// TEST[continued]
779779

780-
The difference here is that instead of passing `q=*` in the URI, we POST a JSON-style query request body to the `_search` API. We'll discuss this JSON query in the next section.
780+
The difference here is that instead of passing `q=*` in the URI, we provide a JSON-style query request body to the `_search` API. We'll discuss this JSON query in the next section.
781781

782782
////
783783
Hidden response just so we can assert that it is indeed the same but don't have

docs/reference/index-modules/similarity.asciidoc

+2-16
Original file line numberDiff line numberDiff line change
@@ -82,20 +82,6 @@ This similarity has the following options:
8282

8383
Type name: `BM25`
8484

85-
[float]
86-
[[classic-similarity]]
87-
==== Classic similarity
88-
89-
The classic similarity that is based on the TF/IDF model. This
90-
similarity has the following option:
91-
92-
`discount_overlaps`::
93-
Determines whether overlap tokens (Tokens with
94-
0 position increment) are ignored when computing norm. By default this
95-
is true, meaning overlap tokens do not count when computing norms.
96-
97-
Type name: `classic`
98-
9985
[float]
10086
[[dfr]]
10187
==== DFR similarity
@@ -541,7 +527,7 @@ PUT /index
541527
"index": {
542528
"similarity": {
543529
"default": {
544-
"type": "classic"
530+
"type": "boolean"
545531
}
546532
}
547533
}
@@ -563,7 +549,7 @@ PUT /index/_settings
563549
"index": {
564550
"similarity": {
565551
"default": {
566-
"type": "classic"
552+
"type": "boolean"
567553
}
568554
}
569555
}

docs/reference/ingest/ingest-node.asciidoc

+1-1
Original file line numberDiff line numberDiff line change
@@ -1341,7 +1341,7 @@ Here is an example of a pipeline specifying custom pattern definitions:
13411341
{
13421342
"grok": {
13431343
"field": "message",
1344-
"patterns": ["my %{FAVORITE_DOG:dog} is colored %{RGB:color}"]
1344+
"patterns": ["my %{FAVORITE_DOG:dog} is colored %{RGB:color}"],
13451345
"pattern_definitions" : {
13461346
"FAVORITE_DOG" : "beagle",
13471347
"RGB" : "RED|GREEN|BLUE"

docs/reference/mapping/params/similarity.asciidoc

+2-7
Original file line numberDiff line numberDiff line change
@@ -44,13 +44,9 @@ PUT my_index
4444
"default_field": { <1>
4545
"type": "text"
4646
},
47-
"classic_field": {
48-
"type": "text",
49-
"similarity": "classic" <2>
50-
},
5147
"boolean_sim_field": {
5248
"type": "text",
53-
"similarity": "boolean" <3>
49+
"similarity": "boolean" <2>
5450
}
5551
}
5652
}
@@ -59,5 +55,4 @@ PUT my_index
5955
--------------------------------------------------
6056
// CONSOLE
6157
<1> The `default_field` uses the `BM25` similarity.
62-
<2> The `classic_field` uses the `classic` similarity (ie TF/IDF).
63-
<3> The `boolean_sim_field` uses the `boolean` similarity.
58+
<2> The `boolean_sim_field` uses the `boolean` similarity.

docs/reference/migration/migrate_7_0/mappings.asciidoc

+13
Original file line numberDiff line numberDiff line change
@@ -24,3 +24,16 @@ the index setting `index.mapping.nested_objects.limit`.
2424
==== The `update_all_types` option has been removed
2525

2626
This option is useless now that all indices have at most one type.
27+
28+
=== The `classic` similarity has been removed
29+
30+
The `classic` similarity relied on coordination factors for scoring to be good
31+
in presence of stopwords in the query. This feature has been removed from
32+
Lucene, which means that the `classic` similarity now produces scores of lower
33+
quality. It is advised to switch to `BM25` instead, which is widely accepted
34+
as a better alternative.
35+
36+
=== Similarities fail when unsupported options are provided
37+
38+
An error will now be thrown when unknown configuration options are provided
39+
to similarities. Such unknown parameters were ignored before.

docs/reference/search/rank-eval.asciidoc

+1-3
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,7 @@
11
[[search-rank-eval]]
22
== Ranking Evaluation API
33

4-
experimental[The ranking evaluation API is experimental and may be changed or removed completely in a future release,
5-
as well as change in non-backwards compatible ways on minor versions updates. Elastic will take a best effort
6-
approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.]
4+
experimental[The ranking evaluation API is experimental and may be changed or removed completely in a future release, as well as change in non-backwards compatible ways on minor versions updates. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.]
75

86
The ranking evaluation API allows to evaluate the quality of ranked search
97
results over a set of typical search queries. Given this set of queries and a
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
2+
// this is just shell gradle file for eclipse to have separate projects for secure-sm src and tests
3+
apply from: '../../build.gradle'
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
2+
// this is just shell gradle file for eclipse to have separate projects for secure-sm src and tests
3+
apply from: '../../build.gradle'
4+
5+
dependencies {
6+
testCompile project(':libs:x-content')
7+
}

modules/analysis-common/src/main/java/org/elasticsearch/analysis/common/TrimTokenFilterFactory.java

+7-1
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,9 @@
2525
import org.elasticsearch.env.Environment;
2626
import org.elasticsearch.index.IndexSettings;
2727
import org.elasticsearch.index.analysis.AbstractTokenFilterFactory;
28+
import org.elasticsearch.index.analysis.MultiTermAwareComponent;
2829

29-
public class TrimTokenFilterFactory extends AbstractTokenFilterFactory {
30+
public class TrimTokenFilterFactory extends AbstractTokenFilterFactory implements MultiTermAwareComponent {
3031

3132
private static final String UPDATE_OFFSETS_KEY = "update_offsets";
3233

@@ -41,4 +42,9 @@ public class TrimTokenFilterFactory extends AbstractTokenFilterFactory {
4142
public TokenStream create(TokenStream tokenStream) {
4243
return new TrimFilter(tokenStream);
4344
}
45+
46+
@Override
47+
public Object getMultiTermComponent() {
48+
return this;
49+
}
4450
}

modules/lang-expression/licenses/lucene-expressions-7.2.1.jar.sha1

-1
This file was deleted.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
38ff5a1f4bcbfb6e1ffacd3263175c2a1ba23e9f

modules/lang-painless/src/main/java/org/elasticsearch/painless/Location.java

+22-34
Original file line numberDiff line numberDiff line change
@@ -27,17 +27,17 @@
2727
public final class Location {
2828
private final String sourceName;
2929
private final int offset;
30-
30+
3131
/**
32-
* Create a new Location
32+
* Create a new Location
3333
* @param sourceName script's name
3434
* @param offset character offset of script element
3535
*/
3636
public Location(String sourceName, int offset) {
3737
this.sourceName = Objects.requireNonNull(sourceName);
3838
this.offset = offset;
3939
}
40-
40+
4141
/**
4242
* Return the script's name
4343
*/
@@ -68,43 +68,31 @@ public RuntimeException createError(RuntimeException exception) {
6868

6969
// This maximum length is theoretically 65535 bytes, but as it's CESU-8 encoded we don't know how large it is in bytes, so be safe
7070
private static final int MAX_NAME_LENGTH = 256;
71-
71+
7272
/** Computes the file name (mostly important for stacktraces) */
73-
public static String computeSourceName(String scriptName, String source) {
73+
public static String computeSourceName(String scriptName) {
7474
StringBuilder fileName = new StringBuilder();
75-
if (scriptName.equals(PainlessScriptEngine.INLINE_NAME)) {
76-
// its an anonymous script, include at least a portion of the source to help identify which one it is
77-
// but don't create stacktraces with filenames that contain newlines or huge names.
75+
// its an anonymous script, include at least a portion of the source to help identify which one it is
76+
// but don't create stacktraces with filenames that contain newlines or huge names.
7877

79-
// truncate to the first newline
80-
int limit = source.indexOf('\n');
81-
if (limit >= 0) {
82-
int limit2 = source.indexOf('\r');
83-
if (limit2 >= 0) {
84-
limit = Math.min(limit, limit2);
85-
}
86-
} else {
87-
limit = source.length();
78+
// truncate to the first newline
79+
int limit = scriptName.indexOf('\n');
80+
if (limit >= 0) {
81+
int limit2 = scriptName.indexOf('\r');
82+
if (limit2 >= 0) {
83+
limit = Math.min(limit, limit2);
8884
}
85+
} else {
86+
limit = scriptName.length();
87+
}
8988

90-
// truncate to our limit
91-
limit = Math.min(limit, MAX_NAME_LENGTH);
92-
fileName.append(source, 0, limit);
89+
// truncate to our limit
90+
limit = Math.min(limit, MAX_NAME_LENGTH);
91+
fileName.append(scriptName, 0, limit);
9392

94-
// if we truncated, make it obvious
95-
if (limit != source.length()) {
96-
fileName.append(" ...");
97-
}
98-
fileName.append(" @ <inline script>");
99-
} else {
100-
// its a named script, just use the name
101-
// but don't trust this has a reasonable length!
102-
if (scriptName.length() > MAX_NAME_LENGTH) {
103-
fileName.append(scriptName, 0, MAX_NAME_LENGTH);
104-
fileName.append(" ...");
105-
} else {
106-
fileName.append(scriptName);
107-
}
93+
// if we truncated, make it obvious
94+
if (limit != scriptName.length()) {
95+
fileName.append(" ...");
10896
}
10997
return fileName.toString();
11098
}

modules/lang-painless/src/main/java/org/elasticsearch/painless/PainlessScript.java

+1-8
Original file line numberDiff line numberDiff line change
@@ -91,14 +91,7 @@ default ScriptException convertToScriptException(Throwable t, Map<String, List<S
9191
scriptStack.add(element.toString());
9292
}
9393
}
94-
// build a name for the script:
95-
final String name;
96-
if (PainlessScriptEngine.INLINE_NAME.equals(getName())) {
97-
name = getSource();
98-
} else {
99-
name = getName();
100-
}
101-
ScriptException scriptException = new ScriptException("runtime error", t, scriptStack, name, PainlessScriptEngine.NAME);
94+
ScriptException scriptException = new ScriptException("runtime error", t, scriptStack, getName(), PainlessScriptEngine.NAME);
10295
for (Map.Entry<String, List<String>> entry : extraMetadata.entrySet()) {
10396
scriptException.addMetadata(entry.getKey(), entry.getValue());
10497
}

modules/lang-painless/src/main/java/org/elasticsearch/painless/PainlessScriptEngine.java

+2-7
Original file line numberDiff line numberDiff line change
@@ -119,11 +119,6 @@ public String getType() {
119119
return NAME;
120120
}
121121

122-
/**
123-
* When a script is anonymous (inline), we give it this name.
124-
*/
125-
static final String INLINE_NAME = "<inline>";
126-
127122
@Override
128123
public <T> T compile(String scriptName, String scriptSource, ScriptContext<T> context, Map<String, String> params) {
129124
Compiler compiler = contextsToCompilers.get(context);
@@ -425,7 +420,7 @@ public Loader run() {
425420
return AccessController.doPrivileged(new PrivilegedAction<Object>() {
426421
@Override
427422
public Object run() {
428-
String name = scriptName == null ? INLINE_NAME : scriptName;
423+
String name = scriptName == null ? source : scriptName;
429424
Constructor<?> constructor = compiler.compile(loader, new MainMethodReserved(), name, source, compilerSettings);
430425

431426
try {
@@ -488,7 +483,7 @@ void compile(Compiler compiler, Loader loader, MainMethodReserved reserved,
488483
AccessController.doPrivileged(new PrivilegedAction<Void>() {
489484
@Override
490485
public Void run() {
491-
String name = scriptName == null ? INLINE_NAME : scriptName;
486+
String name = scriptName == null ? source : scriptName;
492487
compiler.compile(loader, reserved, name, source, compilerSettings);
493488

494489
return null;

modules/lang-painless/src/main/java/org/elasticsearch/painless/antlr/Walker.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -198,7 +198,7 @@ private Walker(ScriptClassInfo scriptClassInfo, MainMethodReserved reserved, Str
198198
this.reserved.push(reserved);
199199
this.debugStream = debugStream;
200200
this.settings = settings;
201-
this.sourceName = Location.computeSourceName(sourceName, sourceText);
201+
this.sourceName = Location.computeSourceName(sourceName);
202202
this.sourceText = sourceText;
203203
this.globals = new Globals(new BitSet(sourceText.length()));
204204
this.definition = definition;

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SSource.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -249,7 +249,7 @@ public void write() {
249249
}
250250
visitor.visit(WriterConstants.CLASS_VERSION, classAccess, className, null,
251251
Type.getType(scriptClassInfo.getBaseClass()).getInternalName(), classInterfaces);
252-
visitor.visitSource(Location.computeSourceName(name, source), null);
252+
visitor.visitSource(Location.computeSourceName(name), null);
253253

254254
// Write the a method to bootstrap def calls
255255
MethodWriter bootstrapDef = new MethodWriter(Opcodes.ACC_STATIC | Opcodes.ACC_VARARGS, DEF_BOOTSTRAP_METHOD, visitor,

modules/parent-join/src/test/java/org/elasticsearch/join/query/HasChildQueryBuilderTests.java

+1-3
Original file line numberDiff line numberDiff line change
@@ -336,9 +336,7 @@ public void testNonDefaultSimilarity() throws Exception {
336336
hasChildQuery(CHILD_DOC, new TermQueryBuilder("custom_string", "value"), ScoreMode.None);
337337
HasChildQueryBuilder.LateParsingQuery query = (HasChildQueryBuilder.LateParsingQuery) hasChildQueryBuilder.toQuery(shardContext);
338338
Similarity expected = SimilarityService.BUILT_IN.get(similarity)
339-
.create(similarity, Settings.EMPTY,
340-
Settings.builder().put(IndexMetaData.SETTING_VERSION_CREATED, Version.CURRENT).build(), null)
341-
.get();
339+
.apply(Settings.EMPTY, Version.CURRENT, null);
342340
assertThat(((PerFieldSimilarityWrapper) query.getSimilarity()).get("custom_string"), instanceOf(expected.getClass()));
343341
}
344342

modules/parent-join/src/test/java/org/elasticsearch/join/query/LegacyHasChildQueryBuilderTests.java

+2-4
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@ protected Collection<Class<? extends Plugin>> getPlugins() {
8787

8888
@Override
8989
protected void initializeAdditionalMappings(MapperService mapperService) throws IOException {
90-
similarity = randomFrom("classic", "BM25");
90+
similarity = randomFrom("boolean", "BM25");
9191
// TODO: use a single type when inner hits have been changed to work with join field,
9292
// this test randomly generates queries with inner hits
9393
mapperService.merge(PARENT_TYPE, new CompressedXContent(Strings.toString(PutMappingRequest.buildFromSimplifiedDef(PARENT_TYPE,
@@ -323,9 +323,7 @@ public void testNonDefaultSimilarity() throws Exception {
323323
hasChildQuery(CHILD_TYPE, new TermQueryBuilder("custom_string", "value"), ScoreMode.None);
324324
HasChildQueryBuilder.LateParsingQuery query = (HasChildQueryBuilder.LateParsingQuery) hasChildQueryBuilder.toQuery(shardContext);
325325
Similarity expected = SimilarityService.BUILT_IN.get(similarity)
326-
.create(similarity, Settings.EMPTY,
327-
Settings.builder().put(IndexMetaData.SETTING_VERSION_CREATED, Version.CURRENT).build(), null)
328-
.get();
326+
.apply(Settings.EMPTY, Version.CURRENT, null);
329327
assertThat(((PerFieldSimilarityWrapper) query.getSimilarity()).get("custom_string"), instanceOf(expected.getClass()));
330328
}
331329

0 commit comments

Comments
 (0)