Skip to content

Refactor SearchRequest to be parsed on the coordinating node #13859

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 99 commits into from
Oct 15, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
a96f14e
new SearchSourceBuilder to use as the search request
colings86 Sep 14, 2015
5623dd2
implemented hashCode, equals and fromXContent in NewSearchSourceBuidler
colings86 Sep 14, 2015
e73052b
cleanup of commented out code
colings86 Sep 14, 2015
eed6ab7
various fixes after a review
colings86 Sep 15, 2015
3c8c6c0
Completed serialisation to/from son and transport protocol and added …
colings86 Sep 15, 2015
9116c1c
Renamed SearchSourceBuilder to OldSearchSourceBuilder and NewSearchSo…
colings86 Sep 16, 2015
e1759b1
cleanup codebase to work with SearchSourceBuilder
colings86 Sep 16, 2015
3cb3514
clean up for methods in SearchRequestBuilder so it works with SearchS…
colings86 Sep 17, 2015
b4dd6a5
fixes to tests
colings86 Sep 22, 2015
aa3d29d
small parsing fix for fields
colings86 Sep 22, 2015
41e6350
muted failing test
colings86 Sep 22, 2015
ab01ec5
SearchRequest.source() now returns a SearchSourceBuilder.
colings86 Sep 22, 2015
b98f7cf
updates to fix some of the NOCOMMITs
colings86 Sep 23, 2015
ac8c172
fix compile errors
colings86 Sep 23, 2015
6c335f4
Parse source, extra-source and template-source on the coordinating node
s1monw Sep 22, 2015
083f28a
remove index from QueryParseContext Constructor
colings86 Sep 23, 2015
a53cf89
Parsing of rescuer and sort ByteReferences.
colings86 Sep 23, 2015
d7429a6
Search service can now fully populate SearchContext from SearchSource…
colings86 Sep 23, 2015
2e3cc50
removed sysout
colings86 Sep 23, 2015
585ed07
unit test fixes
colings86 Sep 23, 2015
52d9664
fix to sort parsing in SearchService
colings86 Sep 23, 2015
394e551
removed sysouts
colings86 Sep 23, 2015
08ae68c
Merge branch 'feature/query-refactoring' into feature/search-request-…
colings86 Sep 23, 2015
63da68e
test fixes
colings86 Sep 23, 2015
85f4c39
more test fixes
colings86 Sep 23, 2015
2de450c
fixes and debugging
colings86 Sep 23, 2015
95a7b6a
Add a late-parsing SearchSoruce wrapper for index warmers
s1monw Sep 23, 2015
04864dc
Impl SearchSource#hashCode|equals|toString
s1monw Sep 23, 2015
7a209cc
more test fixes
colings86 Sep 23, 2015
4996b07
update rest tests to expect default boost to be returned in warmer re…
colings86 Sep 23, 2015
c93eeb8
fix inner_hits parsing in SearchService
s1monw Sep 23, 2015
2dce527
rest test fixes (hopefully)
colings86 Sep 23, 2015
6d307a3
delete by query rest test fix (maybe)
colings86 Sep 23, 2015
3c9e69e
Parse query from deleteByQuery request body
colings86 Sep 23, 2015
a163a0f
fixed script language plugins rest tests to use correct script syntax
colings86 Sep 23, 2015
ce7dde0
fix SearchSourceBuilder.fromXContent to allow sort to be an object
colings86 Sep 23, 2015
f541551
fix lang js rest test (hopefully)
colings86 Sep 23, 2015
cc9519e
correct IndexQueryParserService.parse(QueryBuilder<?>) so that it doe…
colings86 Sep 23, 2015
510d189
Revert "correct IndexQueryParserService.parse(QueryBuilder<?>) so tha…
colings86 Sep 23, 2015
851af9e
fixing up NOCOMMITs and making the IndicesQueryParserService not seri…
colings86 Sep 24, 2015
a9c6e4c
Merge branch 'feature/query-refactoring' into feature/search-request-…
colings86 Sep 24, 2015
67237ba
test fixes
colings86 Sep 24, 2015
bd42d26
text fix
colings86 Sep 24, 2015
35fc73f
fix to make parseSource in SearchService work with source overriding …
colings86 Sep 24, 2015
78d6d29
fix groovy tests - we don't pass json anymore so no need to double es…
s1monw Sep 24, 2015
f3b581a
Added ext section to search request for plugins that extend the fetch…
colings86 Sep 25, 2015
2b94126
removed obselete NOCOMMIT
colings86 Sep 25, 2015
9f08d48
Merge branch 'feature/query-refactoring' into feature/search-request-…
colings86 Sep 25, 2015
f1cf592
Test Fixes
colings86 Sep 25, 2015
8d488c5
Removed delete tests which accidentally got re-added on merge
colings86 Sep 25, 2015
f1da2bb
test fixes
colings86 Sep 25, 2015
4472270
Merge branch 'feature/query-refactoring' into feature/search-request-…
colings86 Sep 25, 2015
05c0031
Merge branch 'master' into feature/search-request-refactoring
s1monw Sep 25, 2015
a21238b
Merge branch 'master' into feature/search-request-refactoring
colings86 Sep 29, 2015
8426c14
fixed compile errors
colings86 Sep 29, 2015
e15736a
fixes for some NORELEASE comments
colings86 Sep 29, 2015
458f87c
removed OldSearchSourceBuilder
colings86 Sep 29, 2015
117d8d2
Merge branch 'master' into feature/search-request-refactoring
colings86 Sep 29, 2015
7fbd565
review comment fixes
colings86 Oct 5, 2015
ec51e8e
Merge branch 'master' into feature/search-request-refactoring
colings86 Oct 5, 2015
e8ef63c
compile fix
colings86 Oct 5, 2015
8149184
test fixes
colings86 Oct 5, 2015
d59e959
rename RestActions.parseQuerySource() to RestActions.urlParamsToQuery…
colings86 Oct 5, 2015
7055a05
test fixes
colings86 Oct 5, 2015
ec93531
fixes
colings86 Oct 5, 2015
9b1d1b2
fixes
colings86 Oct 5, 2015
a3a2432
Merge branch 'master' into feature/search-request-refactoring
colings86 Oct 5, 2015
f89de33
Merge branch 'master' into feature/search-request-refactoring
javanna Oct 6, 2015
f39e23b
pass in QuerySourceBuilder directly to ValidateQueryRequest
javanna Oct 6, 2015
ef286ce
Count api: simplify default handling and toString representation
javanna Oct 6, 2015
80d236e
Revert "Count api: simplify default handling and toString representat…
javanna Oct 6, 2015
669a589
fixed some NORELEASE comments in tests
colings86 Oct 6, 2015
86be9db
Fix some RequestBuilder#toString that produced broken json
javanna Oct 6, 2015
10a5593
more fixes for tests
colings86 Oct 6, 2015
fcc6060
Count api: simplify default handling
javanna Oct 6, 2015
c2fc653
[TEST] move variable assignment one line above
javanna Oct 6, 2015
108f5c5
Count Request: minScore can be null, its getter should be Float rathe…
javanna Oct 6, 2015
1915c74
Merge branch 'master' into feature/search-request-refactoring
javanna Oct 6, 2015
cbcc23b
[TEST] Simplify SearchSourceBuilderTests setup
javanna Oct 6, 2015
9d1c45f
[TEST] Minor SearchSourceBuilderTests infra changes
javanna Oct 6, 2015
3313742
removed defaultRescoreWindow
colings86 Oct 7, 2015
5b23d9a
more test clean ups
colings86 Oct 7, 2015
ee5d284
more clean up
colings86 Oct 7, 2015
a16b025
re-enables InnerHits Tests
colings86 Oct 7, 2015
c69cde4
review fixes
colings86 Oct 8, 2015
42d7bfd
test: add explicit generic type to get around a strange issue where t…
martijnvg Oct 8, 2015
7aaba20
more fixes from review
colings86 Oct 8, 2015
4557d1b
Merge branch 'master' into feature/search-request-refactoring
colings86 Oct 8, 2015
78921e0
test: removed the usage of String.valueOf(...), java 8 type inference…
martijnvg Oct 8, 2015
83771d3
more test fixes
colings86 Oct 8, 2015
e0a78fd
Merge branch 'master' into feature/search-request-refactoring
colings86 Oct 8, 2015
6819224
Merge branch 'master' into feature/search-request-refactoring
colings86 Oct 12, 2015
ca76712
Merge branch 'master' into feature/search-request-refactoring
colings86 Oct 13, 2015
c618f75
Merge branch 'master' into feature/search-request-refactoring
colings86 Oct 15, 2015
63c51b7
review comment fixes
colings86 Oct 15, 2015
8a87448
Revert changes made to IdsQueryBuilder, fixed upstream meanwhile
javanna Oct 15, 2015
0cab4a2
Revert changes made to IndexQueryParserService, fixed upstream meanwhile
javanna Oct 15, 2015
ef5fc72
reverted non-null searchSourceBuilder in searchRequest for now
colings86 Oct 15, 2015
cf3e92a
Merge branch 'master' into feature/search-request-refactoring
colings86 Oct 15, 2015
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ public void readFrom(StreamInput in) throws IOException {
in.readString(),
in.readStringArray(),
in.readOptionalBoolean(),
in.readBytesReference())
in.readBoolean() ? new IndexWarmersMetaData.SearchSource(in) : null)
);
}
warmersMapBuilder.put(key, Collections.unmodifiableList(warmerEntryBuilder));
Expand Down Expand Up @@ -173,7 +173,11 @@ public void writeTo(StreamOutput out) throws IOException {
out.writeString(warmerEntry.name());
out.writeStringArray(warmerEntry.types());
out.writeOptionalBoolean(warmerEntry.requestCache());
out.writeBytesReference(warmerEntry.source());
boolean hasSource = warmerEntry.source() != null;
out.writeBoolean(hasSource);
if (hasSource) {
warmerEntry.source().writeTo(out);
}
}
}
out.writeVInt(mappings.size());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,8 @@
package org.elasticsearch.action.admin.indices.warmer.get;

import com.carrotsearch.hppc.cursors.ObjectObjectCursor;
import org.elasticsearch.Version;

import org.elasticsearch.action.ActionResponse;
import org.elasticsearch.common.bytes.BytesReference;
import org.elasticsearch.common.collect.ImmutableOpenMap;
import org.elasticsearch.common.io.stream.StreamInput;
import org.elasticsearch.common.io.stream.StreamOutput;
Expand Down Expand Up @@ -69,7 +68,10 @@ public void readFrom(StreamInput in) throws IOException {
for (int j = 0; j < valueSize; j++) {
String name = in.readString();
String[] types = in.readStringArray();
BytesReference source = in.readBytesReference();
IndexWarmersMetaData.SearchSource source = null;
if (in.readBoolean()) {
source = new IndexWarmersMetaData.SearchSource(in);
}
Boolean queryCache = null;
queryCache = in.readOptionalBoolean();
warmerEntryBuilder.add(new IndexWarmersMetaData.Entry(
Expand All @@ -94,7 +96,11 @@ public void writeTo(StreamOutput out) throws IOException {
for (IndexWarmersMetaData.Entry warmerEntry : indexEntry.value) {
out.writeString(warmerEntry.name());
out.writeStringArray(warmerEntry.types());
out.writeBytesReference(warmerEntry.source());
boolean hasWarmerSource = warmerEntry != null;
out.writeBoolean(hasWarmerSource);
if (hasWarmerSource) {
warmerEntry.source().writeTo(out);
}
out.writeOptionalBoolean(warmerEntry.requestCache());
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
import org.elasticsearch.common.inject.Inject;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.index.IndexNotFoundException;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.search.warmer.IndexWarmersMetaData;
import org.elasticsearch.threadpool.ThreadPool;
import org.elasticsearch.transport.TransportService;
Expand Down Expand Up @@ -114,11 +115,9 @@ public ClusterState execute(ClusterState currentState) {
MetaData metaData = currentState.metaData();
String[] concreteIndices = indexNameExpressionResolver.concreteIndices(currentState, request.searchRequest().indicesOptions(), request.searchRequest().indices());

BytesReference source = null;
if (request.searchRequest().source() != null && request.searchRequest().source().length() > 0) {
source = request.searchRequest().source();
} else if (request.searchRequest().extraSource() != null && request.searchRequest().extraSource().length() > 0) {
source = request.searchRequest().extraSource();
IndexWarmersMetaData.SearchSource source = null;
if (request.searchRequest().source() != null) {
source = new IndexWarmersMetaData.SearchSource(request.searchRequest().source());
}

// now replace it on the metadata
Expand Down
119 changes: 17 additions & 102 deletions core/src/main/java/org/elasticsearch/action/count/CountRequest.java
Original file line number Diff line number Diff line change
Expand Up @@ -19,140 +19,69 @@

package org.elasticsearch.action.count;

import org.elasticsearch.ElasticsearchGenerationException;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.support.QuerySourceBuilder;
import org.elasticsearch.action.support.broadcast.BroadcastRequest;
import org.elasticsearch.client.Requests;
import org.elasticsearch.common.Nullable;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.bytes.BytesArray;
import org.elasticsearch.common.bytes.BytesReference;
import org.elasticsearch.common.io.stream.StreamInput;
import org.elasticsearch.common.io.stream.StreamOutput;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentFactory;
import org.elasticsearch.common.xcontent.XContentHelper;
import org.elasticsearch.index.query.QueryBuilder;
import org.elasticsearch.search.builder.SearchSourceBuilder;

import java.io.IOException;
import java.util.Arrays;
import java.util.Map;

import static org.elasticsearch.search.internal.SearchContext.DEFAULT_TERMINATE_AFTER;

/**
* A request to count the number of documents matching a specific query. Best created with
* {@link org.elasticsearch.client.Requests#countRequest(String...)}.
* <p>
* The request requires the query source to be set either using {@link #source(QuerySourceBuilder)},
* or {@link #source(byte[])}.
*
* @see CountResponse
* @see org.elasticsearch.client.Client#count(CountRequest)
* @see org.elasticsearch.client.Requests#countRequest(String...)
*/
public class CountRequest extends BroadcastRequest<CountRequest> {

public static final float DEFAULT_MIN_SCORE = -1f;

private float minScore = DEFAULT_MIN_SCORE;

@Nullable
protected String routing;

@Nullable
private String preference;

private BytesReference source;

private String[] types = Strings.EMPTY_ARRAY;

private int terminateAfter = DEFAULT_TERMINATE_AFTER;
private final SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

/**
* Constructs a new count request against the provided indices. No indices provided means it will
* run against all indices.
*/
public CountRequest(String... indices) {
super(indices);
searchSourceBuilder.size(0);
}

/**
* The minimum score of the documents to include in the count.
*/
public float minScore() {
return minScore;
public Float minScore() {
return searchSourceBuilder.minScore();
}

/**
* The minimum score of the documents to include in the count. Defaults to <tt>-1</tt> which means all
* documents will be included in the count.
*/
public CountRequest minScore(float minScore) {
this.minScore = minScore;
return this;
}

/**
* The source to execute.
*/
public BytesReference source() {
return source;
}

/**
* The source to execute.
*/
public CountRequest source(QuerySourceBuilder sourceBuilder) {
this.source = sourceBuilder.buildAsBytes(Requests.CONTENT_TYPE);
return this;
}

/**
* The source to execute in the form of a map.
*/
@SuppressWarnings("unchecked")
public CountRequest source(Map querySource) {
try {
XContentBuilder builder = XContentFactory.contentBuilder(Requests.CONTENT_TYPE);
builder.map(querySource);
return source(builder);
} catch (IOException e) {
throw new ElasticsearchGenerationException("Failed to generate [" + querySource + "]", e);
}
}

public CountRequest source(XContentBuilder builder) {
this.source = builder.bytes();
return this;
}

/**
* The source to execute. It is preferable to use either {@link #source(byte[])}
* or {@link #source(QuerySourceBuilder)}.
*/
public CountRequest source(String querySource) {
this.source = new BytesArray(querySource);
this.searchSourceBuilder.minScore(minScore);
return this;
}

/**
* The source to execute.
*/
public CountRequest source(byte[] querySource) {
return source(querySource, 0, querySource.length);
}

/**
* The source to execute.
* The query to execute
*/
public CountRequest source(byte[] querySource, int offset, int length) {
return source(new BytesArray(querySource, offset, length));
}

public CountRequest source(BytesReference querySource) {
this.source = querySource;
public CountRequest query(QueryBuilder<?> queryBuilder) {
this.searchSourceBuilder.query(queryBuilder);
return this;
}

Expand Down Expand Up @@ -207,15 +136,12 @@ public String preference() {
* Upon reaching <code>terminateAfter</code> counts, the count request will early terminate
*/
public CountRequest terminateAfter(int terminateAfterCount) {
if (terminateAfterCount <= 0) {
throw new IllegalArgumentException("terminateAfter must be > 0");
}
this.terminateAfter = terminateAfterCount;
this.searchSourceBuilder.terminateAfter(terminateAfterCount);
return this;
}

public int terminateAfter() {
return this.terminateAfter;
return this.searchSourceBuilder.terminateAfter();
}

@Override
Expand All @@ -230,31 +156,20 @@ public void writeTo(StreamOutput out) throws IOException {

@Override
public String toString() {
String sSource = "_na_";
try {
sSource = XContentHelper.convertToJson(source, false);
} catch (Exception e) {
// ignore
}
return "[" + Arrays.toString(indices) + "]" + Arrays.toString(types) + ", source[" + sSource + "]";
return "count request indices:" + Arrays.toString(indices) +
", types:" + Arrays.toString(types) +
", routing: " + routing +
", preference: " + preference +
", source:" + searchSourceBuilder.toString();
}

public SearchRequest toSearchRequest() {
SearchRequest searchRequest = new SearchRequest(indices());
searchRequest.source(searchSourceBuilder);
searchRequest.indicesOptions(indicesOptions());
searchRequest.types(types());
searchRequest.routing(routing());
searchRequest.preference(preference());
searchRequest.source(source());
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.size(0);
if (minScore() != DEFAULT_MIN_SCORE) {
searchSourceBuilder.minScore(minScore());
}
if (terminateAfter() != DEFAULT_TERMINATE_AFTER) {
searchSourceBuilder.terminateAfter(terminateAfter());
}
searchRequest.extraSource(searchSourceBuilder);
return searchRequest;
}
}
Loading