Merge branch 'master' into feature/eql

Author: costin

docs/reference/aggregations/metrics/percentile-rank-aggregation.asciidoc

@@ -183,7 +183,7 @@ number of significant digits). This means that if data is recorded with values f
 microseconds) in a histogram set to 3 significant digits, it will maintain a value resolution of 1 microsecond for values up to
 1 millisecond and 3.6 seconds (or better) for the maximum tracked value (1 hour).
 
-The HDR Histogram can be used by specifying the `method` parameter in the request:
+The HDR Histogram can be used by specifying the `hdr` object in the request:
 
 [source,console]
 --------------------------------------------------

docs/reference/ml/anomaly-detection/aggregations.asciidoc

@@ -11,13 +11,28 @@ distributes these calculations across your cluster. You can then feed this
 aggregated data into the {ml-features} instead of raw results, which
 reduces the volume of data that must be considered while detecting anomalies.
 
-There are some limitations to using aggregations in {dfeeds}, however.
-Your aggregation must include a `date_histogram` aggregation, which in turn must
-contain a `max` aggregation on the time field. This requirement ensures that the
-aggregated data is a time series and the timestamp of each bucket is the time
-of the last record in the bucket. If you use a terms aggregation and the
-cardinality of a term is high, then the aggregation might not be effective and
-you might want to just use the default search and scroll behavior.
+TIP: If you use a terms aggregation and the cardinality of a term is high, the
+aggregation might not be effective and you might want to just use the default
+search and scroll behavior.
+
+There are some limitations to using aggregations in {dfeeds}. Your aggregation
+must include a `date_histogram` aggregation, which in turn must contain a `max`
+aggregation on the time field. This requirement ensures that the aggregated data
+is a time series and the timestamp of each bucket is the time of the last record
+in the bucket.
+
+You must also consider the interval of the date histogram aggregation carefully.
+The bucket span of your {anomaly-job} must be divisible by the value of the
+`calendar_interval` or `fixed_interval` in your aggregation (with no remainder).
+If you specify a `frequency` for your {dfeed}, it must also be divisible by this
+interval.
+
+TIP: As a rule of thumb, if your detectors use <<ml-metric-functions,metric>> or
+<<ml-sum-functions,sum>> analytical functions, set the date histogram
+aggregation interval to a tenth of the bucket span. This suggestion creates
+finer, more granular time buckets, which are ideal for this type of analysis. If
+your detectors use <<ml-count-functions,count>> or <<ml-rare-functions,rare>>
+functions, set the interval to the same value as the bucket span.
 
 When you create or update an {anomaly-job}, you can include the names of
 aggregations, for example:
@@ -143,9 +158,9 @@ pipeline aggregation to find the first order derivative of the counter
 ----------------------------------
 // NOTCONSOLE
 
-{dfeeds-cap} not only supports multi-bucket aggregations, but also single bucket aggregations.
-The following shows two `filter` aggregations, each gathering the number of unique entries for
-the `error` field.
+{dfeeds-cap} not only supports multi-bucket aggregations, but also single bucket
+aggregations. The following shows two `filter` aggregations, each gathering the
+number of unique entries for the `error` field.
 
 [source,js]
 ----------------------------------
@@ -225,14 +240,15 @@ When you define an aggregation in a {dfeed}, it must have the following form:
 ----------------------------------
 // NOTCONSOLE
 
-The top level aggregation must be either a {ref}/search-aggregations-bucket.html[Bucket Aggregation]
-containing as single sub-aggregation that is a `date_histogram` or the top level aggregation
-is the required `date_histogram`. There must be exactly 1 `date_histogram` aggregation.
+The top level aggregation must be either a
+{ref}/search-aggregations-bucket.html[bucket aggregation] containing as single
+sub-aggregation that is a `date_histogram` or the top level aggregation is the
+required `date_histogram`. There must be exactly 1 `date_histogram` aggregation.
 For more information, see
-{ref}/search-aggregations-bucket-datehistogram-aggregation.html[Date Histogram Aggregation].
+{ref}/search-aggregations-bucket-datehistogram-aggregation.html[Date histogram aggregation].
 
-NOTE: The `time_zone` parameter in the date histogram aggregation must be set to `UTC`,
-which is the default value.
+NOTE: The `time_zone` parameter in the date histogram aggregation must be set to
+`UTC`, which is the default value.
 
 Each histogram bucket has a key, which is the bucket start time. This key cannot
 be used for aggregations in {dfeeds}, however, because they need to know the
@@ -269,16 +285,9 @@ By default, {es} limits the maximum number of terms returned to 10000. For high
 cardinality fields, the query might not run. It might return errors related to
 circuit breaking exceptions that indicate that the data is too large. In such
 cases, do not use aggregations in your {dfeed}. For more
-information, see {ref}/search-aggregations-bucket-terms-aggregation.html[Terms Aggregation].
-
-You can also optionally specify multiple sub-aggregations.
-The sub-aggregations are aggregated for the buckets that were created by their
-parent aggregation. For more information, see
-{ref}/search-aggregations.html[Aggregations].
+information, see
+{ref}/search-aggregations-bucket-terms-aggregation.html[Terms aggregation].
 
-TIP: If your detectors use metric or sum analytical functions, set the
-`interval` of the date histogram aggregation to a tenth of the `bucket_span`
-that was defined in the job. This suggestion creates finer, more granular time
-buckets, which are ideal for this type of analysis. If your detectors use count
-or rare functions, set `interval` to the same value as `bucket_span`. For more
-information about analytical functions, see <<ml-functions>>.
+You can also optionally specify multiple sub-aggregations. The sub-aggregations
+are aggregated for the buckets that were created by their parent aggregation.
+For more information, see {ref}/search-aggregations.html[Aggregations].

docs/reference/ml/anomaly-detection/apis/put-datafeed.asciidoc

@@ -26,7 +26,12 @@ cluster privileges to use this API. See
 [[ml-put-datafeed-desc]]
 ==== {api-description-title}
 
-You can associate only one {dfeed} to each {anomaly-job}.
+{ml-docs}/ml-dfeeds.html[{dfeeds-cap}] retrieve data from {es} for analysis by
+an {anomaly-job}. You can associate only one {dfeed} to each {anomaly-job}.
+
+The {dfeed} contains a query that runs at a defined interval (`frequency`). If
+you are concerned about delayed data, you can add a delay (`query_delay`) at
+each interval. See {ml-docs}/ml-delayed-data-detection.html[Handling delayed data].
 
 [IMPORTANT]
 ====
@@ -64,11 +69,6 @@ include::{docdir}/ml/ml-shared.asciidoc[tag=delayed-data-check-config]
 `frequency`::
 (Optional, <<time-units, time units>>)
 include::{docdir}/ml/ml-shared.asciidoc[tag=frequency]
-+
---
-To learn more about the relationship between time related settings, see 
-<<ml-put-datafeed-time-related-settings>>.
---
 
 `indices`::
 (Required, array)
@@ -89,11 +89,6 @@ include::{docdir}/ml/ml-shared.asciidoc[tag=query]
 `query_delay`::
 (Optional, <<time-units, time units>>)
 include::{docdir}/ml/ml-shared.asciidoc[tag=query-delay]
-+
---
-To learn more about the relationship between time related settings, see 
-<<ml-put-datafeed-time-related-settings>>.
---
 
 `script_fields`::
 (Optional, object)
@@ -103,20 +98,6 @@ include::{docdir}/ml/ml-shared.asciidoc[tag=script-fields]
 (Optional, unsigned integer)
 include::{docdir}/ml/ml-shared.asciidoc[tag=scroll-size]
 
-
-[[ml-put-datafeed-time-related-settings]]
-===== Interaction between time-related settings
-
-Time-related settings have the following relationships:
-
-* Queries run at `query_delay` after the end of 
-  each `frequency`.
-  
-* When `frequency` is shorter than `bucket_span` of the associated job, interim 
-  results for the last (partial) bucket are written, and then overwritten by the 
-  full bucket results eventually.
-
-
 [[ml-put-datafeed-example]]
 ==== {api-examples-title}
 

docs/reference/ml/anomaly-detection/apis/put-job.asciidoc

@@ -49,11 +49,6 @@ include::{docdir}/ml/ml-shared.asciidoc[tag=analysis-config]
 `analysis_config`.`bucket_span`:::
 (<<time-units,time units>>)
 include::{docdir}/ml/ml-shared.asciidoc[tag=bucket-span]
-+
---
-To learn more about the relationship between time related settings, see 
-<<ml-put-datafeed-time-related-settings>>.
---
 
 `analysis_config`.`categorization_field_name`:::
 (string)

docs/reference/ml/ml-shared.asciidoc

@@ -1,6 +1,6 @@
 tag::aggregations[]
 If set, the {dfeed} performs aggregation searches. Support for aggregations is
-limited and should only be used with low cardinality data. For more information,
+limited and should be used only with low cardinality data. For more information,
 see
 {ml-docs}/ml-configuring-aggregation.html[Aggregating data for faster performance].
 end::aggregations[]
@@ -149,8 +149,10 @@ end::background-persist-interval[]
 
 tag::bucket-span[]
 The size of the interval that the analysis is aggregated into, typically between
-`5m` and `1h`. The default value is `5m`. For more information about time units,
-see <<time-units>>.
+`5m` and `1h`. The default value is `5m`. If the {anomaly-job} uses a {dfeed}
+with {ml-docs}/ml-configuring-aggregation.html[aggregations], this value must be
+divisible by the interval of the date histogram aggregation. For more
+information, see {ml-docs}/ml-buckets.html[Buckets]. 
 end::bucket-span[]
 
 tag::bucket-span-results[]
@@ -605,7 +607,10 @@ tag::frequency[]
 The interval at which scheduled queries are made while the {dfeed} runs in real
 time. The default value is either the bucket span for short bucket spans, or,
 for longer bucket spans, a sensible fraction of the bucket span. For example:
-`150s`.
+`150s`. When `frequency` is shorter than the bucket span, interim results for
+the last (partial) bucket are written then eventually overwritten by the full
+bucket results. If the {dfeed} uses aggregations, this value must be divisible
+by the interval of the date histogram aggregation.
 end::frequency[]
 
 tag::from[]
@@ -949,7 +954,8 @@ The number of seconds behind real time that data is queried. For example, if
 data from 10:04 a.m. might not be searchable in {es} until 10:06 a.m., set this
 property to 120 seconds. The default value is randomly selected between `60s`
 and `120s`. This randomness improves the query performance when there are
-multiple jobs running on the same node.
+multiple jobs running on the same node. For more information, see
+{ml-docs}/ml-delayed-data-detection.html[Handling delayed data].
 end::query-delay[]
 
 tag::renormalization-window-days[]

server/src/main/java/org/elasticsearch/index/engine/CombinedDocValues.java

@@ -47,6 +47,7 @@ final class CombinedDocValues {
     long docVersion(int segmentDocId) throws IOException {
         assert versionDV.docID() < segmentDocId;
         if (versionDV.advanceExact(segmentDocId) == false) {
+            assert false : "DocValues for field [" + VersionFieldMapper.NAME + "] is not found";
             throw new IllegalStateException("DocValues for field [" + VersionFieldMapper.NAME + "] is not found");
         }
         return versionDV.longValue();
@@ -55,19 +56,18 @@ long docVersion(int segmentDocId) throws IOException {
     long docSeqNo(int segmentDocId) throws IOException {
         assert seqNoDV.docID() < segmentDocId;
         if (seqNoDV.advanceExact(segmentDocId) == false) {
+            assert false : "DocValues for field [" + SeqNoFieldMapper.NAME + "] is not found";
             throw new IllegalStateException("DocValues for field [" + SeqNoFieldMapper.NAME + "] is not found");
         }
         return seqNoDV.longValue();
     }
 
     long docPrimaryTerm(int segmentDocId) throws IOException {
-        if (primaryTermDV == null) {
-            return -1L;
-        }
+        // We exclude non-root nested documents when querying changes, every returned document must have primary term.
         assert primaryTermDV.docID() < segmentDocId;
-        // Use -1 for docs which don't have primary term. The caller considers those docs as nested docs.
         if (primaryTermDV.advanceExact(segmentDocId) == false) {
-            return -1;
+            assert false : "DocValues for field [" + SeqNoFieldMapper.PRIMARY_TERM_NAME + "] is not found";
+            throw new IllegalStateException("DocValues for field [" + SeqNoFieldMapper.PRIMARY_TERM_NAME + "] is not found");
         }
         return primaryTermDV.longValue();
     }

server/src/main/java/org/elasticsearch/index/engine/InternalEngine.java

@@ -38,6 +38,8 @@
 import org.apache.lucene.index.ShuffleForcedMergePolicy;
 import org.apache.lucene.index.SoftDeletesRetentionMergePolicy;
 import org.apache.lucene.index.Term;
+import org.apache.lucene.search.BooleanClause;
+import org.apache.lucene.search.BooleanQuery;
 import org.apache.lucene.search.DocIdSetIterator;
 import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.Query;
@@ -63,6 +65,7 @@
 import org.elasticsearch.common.lucene.LoggerInfoStream;
 import org.elasticsearch.common.lucene.Lucene;
 import org.elasticsearch.common.lucene.index.ElasticsearchDirectoryReader;
+import org.elasticsearch.common.lucene.search.Queries;
 import org.elasticsearch.common.lucene.uid.Versions;
 import org.elasticsearch.common.lucene.uid.VersionsAndSeqNoResolver;
 import org.elasticsearch.common.lucene.uid.VersionsAndSeqNoResolver.DocIdAndSeqNo;
@@ -2731,8 +2734,12 @@ private boolean assertMaxSeqNoOfUpdatesIsAdvanced(Term id, long seqNo, boolean a
     private void restoreVersionMapAndCheckpointTracker(DirectoryReader directoryReader) throws IOException {
         final IndexSearcher searcher = new IndexSearcher(directoryReader);
         searcher.setQueryCache(null);
-        final Query query = LongPoint.newRangeQuery(SeqNoFieldMapper.NAME, getPersistedLocalCheckpoint() + 1, Long.MAX_VALUE);
-        final Weight weight = searcher.createWeight(query, ScoreMode.COMPLETE_NO_SCORES, 1.0f);
+        final Query query = new BooleanQuery.Builder()
+            .add(LongPoint.newRangeQuery(
+                    SeqNoFieldMapper.NAME, getPersistedLocalCheckpoint() + 1, Long.MAX_VALUE), BooleanClause.Occur.MUST)
+            .add(Queries.newNonNestedFilter(), BooleanClause.Occur.MUST) // exclude non-root nested documents
+            .build();
+        final Weight weight = searcher.createWeight(searcher.rewrite(query), ScoreMode.COMPLETE_NO_SCORES, 1.0f);
         for (LeafReaderContext leaf : directoryReader.leaves()) {
             final Scorer scorer = weight.scorer(leaf);
             if (scorer == null) {
@@ -2744,9 +2751,6 @@ private void restoreVersionMapAndCheckpointTracker(DirectoryReader directoryRead
             int docId;
             while ((docId = iterator.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS) {
                 final long primaryTerm = dv.docPrimaryTerm(docId);
-                if (primaryTerm == -1L) {
-                    continue; // skip children docs which do not have primary term
-                }
                 final long seqNo = dv.docSeqNo(docId);
                 localCheckpointTracker.markSeqNoAsProcessed(seqNo);
                 localCheckpointTracker.markSeqNoAsPersisted(seqNo);

server/src/main/java/org/elasticsearch/index/engine/LuceneChangesSnapshot.java

@@ -24,6 +24,8 @@
 import org.apache.lucene.index.LeafReaderContext;
 import org.apache.lucene.index.NumericDocValues;
 import org.apache.lucene.index.Term;
+import org.apache.lucene.search.BooleanClause;
+import org.apache.lucene.search.BooleanQuery;
 import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.Query;
 import org.apache.lucene.search.ScoreDoc;
@@ -33,6 +35,7 @@
 import org.apache.lucene.util.ArrayUtil;
 import org.elasticsearch.common.bytes.BytesReference;
 import org.elasticsearch.common.lucene.Lucene;
+import org.elasticsearch.common.lucene.search.Queries;
 import org.elasticsearch.core.internal.io.IOUtils;
 import org.elasticsearch.index.fieldvisitor.FieldsVisitor;
 import org.elasticsearch.index.mapper.IdFieldMapper;
@@ -210,7 +213,10 @@ private void fillParallelArray(ScoreDoc[] scoreDocs, ParallelArray parallelArray
     }
 
     private TopDocs searchOperations(ScoreDoc after) throws IOException {
-        final Query rangeQuery = LongPoint.newRangeQuery(SeqNoFieldMapper.NAME, Math.max(fromSeqNo, lastSeenSeqNo), toSeqNo);
+        final Query rangeQuery = new BooleanQuery.Builder()
+            .add(LongPoint.newRangeQuery(SeqNoFieldMapper.NAME, Math.max(fromSeqNo, lastSeenSeqNo), toSeqNo), BooleanClause.Occur.MUST)
+            .add(Queries.newNonNestedFilter(), BooleanClause.Occur.MUST) // exclude non-root nested documents
+            .build();
         final Sort sortedBySeqNo = new Sort(new SortField(SeqNoFieldMapper.NAME, SortField.Type.LONG));
         return indexSearcher.searchAfter(after, rangeQuery, searchBatchSize, sortedBySeqNo);
     }
@@ -219,11 +225,7 @@ private Translog.Operation readDocAsOp(int docIndex) throws IOException {
         final LeafReaderContext leaf = parallelArray.leafReaderContexts[docIndex];
         final int segmentDocID = scoreDocs[docIndex].doc - leaf.docBase;
         final long primaryTerm = parallelArray.primaryTerm[docIndex];
-        // We don't have to read the nested child documents - those docs don't have primary terms.
-        if (primaryTerm == -1) {
-            skippedOperations++;
-            return null;
-        }
+        assert primaryTerm > 0 : "nested child document must be excluded";
         final long seqNo = parallelArray.seqNo[docIndex];
         // Only pick the first seen seq#
         if (seqNo == lastSeenSeqNo) {

server/src/main/java/org/elasticsearch/index/mapper/DocumentParser.java

@@ -24,6 +24,7 @@
 import org.elasticsearch.ElasticsearchParseException;
 import org.elasticsearch.common.Strings;
 import org.elasticsearch.common.collect.Tuple;
+import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.time.DateFormatter;
 import org.elasticsearch.common.xcontent.LoggingDeprecationHandler;
 import org.elasticsearch.common.xcontent.XContentHelper;
@@ -417,21 +418,23 @@ private static void innerParseObject(ParseContext context, ObjectMapper mapper,
     private static void nested(ParseContext context, ObjectMapper.Nested nested) {
         ParseContext.Document nestedDoc = context.doc();
         ParseContext.Document parentDoc = nestedDoc.getParent();
+        Settings settings = context.indexSettings().getSettings();
         if (nested.isIncludeInParent()) {
-            addFields(nestedDoc, parentDoc);
+            addFields(settings, nestedDoc, parentDoc);
         }
         if (nested.isIncludeInRoot()) {
             ParseContext.Document rootDoc = context.rootDoc();
             // don't add it twice, if its included in parent, and we are handling the master doc...
             if (!nested.isIncludeInParent() || parentDoc != rootDoc) {
-                addFields(nestedDoc, rootDoc);
+                addFields(settings, nestedDoc, rootDoc);
             }
         }
     }
 
-    private static void addFields(ParseContext.Document nestedDoc, ParseContext.Document rootDoc) {
+    private static void addFields(Settings settings, ParseContext.Document nestedDoc, ParseContext.Document rootDoc) {
+        String nestedPathFieldName = NestedPathFieldMapper.name(settings);
         for (IndexableField field : nestedDoc.getFields()) {
-            if (!field.name().equals(TypeFieldMapper.NAME)) {
+            if (field.name().equals(nestedPathFieldName) == false) {
                 rootDoc.add(field);
             }
         }
@@ -457,10 +460,7 @@ private static ParseContext nestedContext(ParseContext context, ObjectMapper map
             throw new IllegalStateException("The root document of a nested document should have an _id field");
         }
 
-        // the type of the nested doc starts with __, so we can identify that its a nested one in filters
-        // note, we don't prefix it with the type of the doc since it allows us to execute a nested query
-        // across types (for example, with similar nested objects)
-        nestedDoc.add(new Field(TypeFieldMapper.NAME, mapper.nestedTypePathAsString(), TypeFieldMapper.Defaults.FIELD_TYPE));
+        nestedDoc.add(NestedPathFieldMapper.field(context.indexSettings().getSettings(), mapper.nestedTypePath()));
         return context;
     }