Limit the analyzed text for highlighting (elastic#27934)

- Introduce index level setting "index.highlight.max_analyzed_offset"
to control the max number of character to be analyzed for highlighting
- Make this setting to be unset by default (equal to -1)
- Issue a deprecation warning if setting is unset and analysis is required
 on a text larger than ES v.7.x max setting (10000)
- Throw IllegalArgumentException is setting is set by a user, and
analysis is required on a text larger than the user's set value.

Closes elastic#27517

Adding validator for index.highlight.max_analyzed_offset setting

Author: mayya-sharipova

docs/reference/index-modules.asciidoc

@@ -186,7 +186,7 @@ specific index module:
 
      The maximum number of characters that will be analyzed for a highlight request.
      This setting is only applicable when highlighting is requested on a text that was indexed without offsets or term vectors.
-     Defaults to `10000`.
+     By default this settings is unset in 6.x, defaults to `-1`.
 
  `index.max_terms_count`::
 

docs/reference/migration/migrate_6_0/analysis.asciidoc

@@ -16,6 +16,8 @@ created in 5.x.
 Highlighting a text that was indexed without offsets or term vectors,
 requires analysis of this text in memory real time during the search request.
 For large texts this analysis may take substantial amount of time and memory.
-To protect against this, the maximum number of characters that will be analyzed has been
-limited to 10000. This default limit can be changed
-for a particular index with the index setting `index.highlight.max_analyzed_offset`.
+To protect against this, the maximum number of characters that to be analyzed will be
+limited to 10000 in the next major Elastic version. For this version, by default the limit
+is not set. A deprecation warning will be issued when an analyzed text exceeds 10000.
+ The limit can be set for a particular index with the index setting
+`index.highlight.max_analyzed_offset`.

docs/reference/search/request/highlighting.asciidoc

@@ -106,9 +106,9 @@ needs highlighting. The `plain` highlighter always uses plain highlighting.
 
 [WARNING]
 Plain highlighting for large texts may require substantial amount of time and memory.
-To protect against this, the maximum number of text characters that will be analyzed has been
-limited to 10000. This default limit can be changed
-for a particular index with the index setting `index.highlight.max_analyzed_offset`.
+To protect against this, the maximum number of text characters to be analyzed will be
+limited to 10000 in the next major Elastic version. The default limit is not set for this version,
+but can be set for a particular index with the index setting `index.highlight.max_analyzed_offset`.
 
 [[highlighting-settings]]
 ==== Highlighting Settings

rest-api-spec/src/main/resources/rest-api-spec/test/search.highlight/30_max_analyzed_offset.yml

@@ -33,27 +33,24 @@ setup:
   - skip:
       version: " - 6.1.99"
       reason: index.highlight.max_analyzed_offset setting has been added in 6.2
-      features: "warnings"
   - do:
+      catch: bad_request
       search:
           index: test1
           body: {"query" : {"match" : {"field1" : "fox"}}, "highlight" : {"type" : "unified", "fields" : {"field1" : {}}}}
-      warnings:
-          - Deprecated large text to be analyzed for highlighting! The length has exceeded the allowed maximum of [10]. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!
-
+  - match: { error.root_cause.0.type: "illegal_argument_exception" }
 
 ---
 "Plain highlighter on a field WITHOUT OFFSETS exceeding index.highlight.max_analyzed_offset should FAIL":
   - skip:
       version: " - 6.1.99"
       reason: index.highlight.max_analyzed_offset setting has been added in 6.2
-      features: "warnings"
   - do:
+      catch: bad_request
       search:
           index: test1
           body: {"query" : {"match" : {"field1" : "fox"}}, "highlight" : {"type" : "plain", "fields" : {"field1" : {}}}}
-      warnings:
-          - Deprecated large text to be analyzed for highlighting! The length has exceeded the allowed maximum of [10]. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors, and highlighting with unified or fvh highlighter is recommended!
+  - match: { error.root_cause.0.type: "illegal_argument_exception" }
 
 ---
 "Unified highlighter on a field WITH OFFSETS exceeding index.highlight.max_analyzed_offset should SUCCEED":
@@ -72,10 +69,9 @@ setup:
   - skip:
       version: " - 6.1.99"
       reason: index.highlight.max_analyzed_offset setting has been added in 6.2
-      features: "warnings"
   - do:
+      catch: bad_request
       search:
           index: test1
           body: {"query" : {"match" : {"field2" : "fox"}}, "highlight" : {"type" : "plain", "fields" : {"field2" : {}}}}
-      warnings:
-          - Deprecated large text to be analyzed for highlighting! The length has exceeded the allowed maximum of [10]. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors, and highlighting with unified or fvh highlighter is recommended!
+  - match: { error.root_cause.0.type: "illegal_argument_exception" }

server/src/main/java/org/apache/lucene/search/uhighlight/CustomUnifiedHighlighter.java

@@ -128,11 +128,21 @@ public Snippet[] highlightField(String field, Query query, int docId, int maxPas
     @Override
     protected List<CharSequence[]> loadFieldValues(String[] fields, DocIdSetIterator docIter,
                                                    int cacheCharsThreshold) throws IOException {
-        if ((offsetSource == OffsetSource.ANALYSIS) && (fieldValue.length() > maxAnalyzedOffset)) {
+        // Issue deprecation warning if maxAnalyzedOffset is not set, and field length > default setting for 7.0
+        final int defaultMaxAnalyzedOffset7 = 10000;
+        if ((offsetSource == OffsetSource.ANALYSIS) &&  (maxAnalyzedOffset == -1) && (fieldValue.length() > defaultMaxAnalyzedOffset7)) {
             DeprecationLogger deprecationLogger = new DeprecationLogger(Loggers.getLogger(CustomUnifiedHighlighter.class));
             deprecationLogger.deprecated(
-                "Deprecated large text to be analyzed for highlighting! The length has exceeded the allowed maximum of [" +
-                    maxAnalyzedOffset + "]. " + "This maximum can be set by changing the [" +
+                "The length of text to be analyzed for highlighting [" + fieldValue.length() +
+                    "] exceeded the allowed maximum of [" + defaultMaxAnalyzedOffset7 + "] set for the next major Elastic version. " +
+                    "For large texts, indexing with offsets or term vectors is recommended!");
+        }
+        // Throw an error if maxAnalyzedOffset is explicitly set by the user, and field length > maxAnalyzedOffset
+        if ((offsetSource == OffsetSource.ANALYSIS) &&  (maxAnalyzedOffset > 0) && (fieldValue.length() > maxAnalyzedOffset)) {
+            // maxAnalyzedOffset is not set by user
+            throw new IllegalArgumentException(
+                "The length of text to be analyzed for highlighting [" + fieldValue.length() +
+                    "] exceeded the allowed maximum of [" + maxAnalyzedOffset + "]. This maximum can be set by changing the [" +
                     IndexSettings.MAX_ANALYZED_OFFSET_SETTING.getKey() + "] index level setting. " +
                     "For large texts, indexing with offsets or term vectors is recommended!");
         }

server/src/main/java/org/elasticsearch/index/IndexSettings.java

@@ -126,11 +126,11 @@ public final class IndexSettings {
      * A setting describing the maximum number of characters that will be analyzed for a highlight request.
      * This setting is only applicable when highlighting is requested on a text that was indexed without
      * offsets or term vectors.
-     * The default maximum of 10000 characters is defensive as for highlighting larger texts,
-     * indexing with offsets or term vectors is recommended.
+     * This setting is defensive as for highlighting larger texts, indexing with offsets or term vectors is recommended.
+     * For 6.x the default value is not set or equals to -1.
      */
     public static final Setting<Integer> MAX_ANALYZED_OFFSET_SETTING =
-        Setting.intSetting("index.highlight.max_analyzed_offset", 10000, 1, Property.Dynamic, Property.IndexScope);
+        Setting.intSetting("index.highlight.max_analyzed_offset", -1, -1, Property.Dynamic, Property.IndexScope);
 
 
     /**
@@ -727,7 +727,13 @@ private void setMaxDocvalueFields(int maxDocvalueFields) {
      */
     public int getHighlightMaxAnalyzedOffset() { return this.maxAnalyzedOffset; }
 
-    private void setHighlightMaxAnalyzedOffset(int maxAnalyzedOffset) { this.maxAnalyzedOffset = maxAnalyzedOffset; }
+    private void setHighlightMaxAnalyzedOffset(int maxAnalyzedOffset) {
+        if (maxAnalyzedOffset < 1) {
+            throw new IllegalArgumentException(
+                "[" + MAX_ANALYZED_OFFSET_SETTING.getKey() + "] must be >= 1");
+        }
+        this.maxAnalyzedOffset = maxAnalyzedOffset;
+    }
 
     /**
      *  Returns the maximum number of terms that can be used in a Terms Query request

server/src/main/java/org/elasticsearch/search/fetch/subphase/highlight/PlainHighlighter.java

@@ -110,17 +110,26 @@ public HighlightField highlight(HighlighterContext highlighterContext) {
 
         try {
             textsToHighlight = HighlightUtils.loadFieldValues(field, mapper, context, hitContext);
-
+            final int defaultMaxAnalyzedOffset7 = 10000;
             for (Object textToHighlight : textsToHighlight) {
                 String text = convertFieldValue(mapper.fieldType(), textToHighlight);
-                if (text.length() > maxAnalyzedOffset) {
+
+                // Issue deprecation warning if maxAnalyzedOffset is not set, and text length > default setting for 7.0
+                if ((maxAnalyzedOffset == -1) && (text.length() > defaultMaxAnalyzedOffset7)) {
                     DeprecationLogger deprecationLogger = new DeprecationLogger(Loggers.getLogger(PlainHighlighter.class));
                     deprecationLogger.deprecated(
-                        "Deprecated large text to be analyzed for highlighting! The length has exceeded the allowed maximum of [" +
-                            maxAnalyzedOffset + "]. " + "This maximum can be set by changing the [" +
+                        "The length of text to be analyzed for highlighting [" + text.length() + "] exceeded the allowed maximum of [" +
+                            defaultMaxAnalyzedOffset7 + "] set for the next major Elastic version. " +
+                            "For large texts, indexing with offsets or term vectors is recommended!");
+                }
+                // Throw an error if maxAnalyzedOffset is explicitly set by the user, and text length > maxAnalyzedOffset
+                if ((maxAnalyzedOffset > 0) && (text.length() > maxAnalyzedOffset)) {
+                    // maxAnalyzedOffset is not set by user
+                    throw new IllegalArgumentException(
+                        "The length of text to be analyzed for highlighting [" + text.length() +
+                            "] exceeded the allowed maximum of [" + maxAnalyzedOffset + "]. This maximum can be set by changing the [" +
                             IndexSettings.MAX_ANALYZED_OFFSET_SETTING.getKey() + "] index level setting. " +
-                            "For large texts, indexing with offsets or term vectors, and highlighting with unified or " +
-                            "fvh highlighter is recommended!");
+                            "For large texts, indexing with offsets or term vectors is recommended!");
                 }
 
                 try (TokenStream tokenStream = analyzer.tokenStream(mapper.fieldType().name(), text)) {

server/src/test/java/org/apache/lucene/search/uhighlight/CustomUnifiedHighlighterTests.java

@@ -79,7 +79,7 @@ private void assertHighlightOneDoc(String fieldName, String[] inputs, Analyzer a
         String rawValue = Strings.arrayToDelimitedString(inputs, String.valueOf(MULTIVAL_SEP_CHAR));
         CustomUnifiedHighlighter highlighter = new CustomUnifiedHighlighter(searcher, analyzer, null,
                 new CustomPassageFormatter("<b>", "</b>", new DefaultEncoder()), locale,
-                breakIterator, rawValue, noMatchSize, 10000);
+                breakIterator, rawValue, noMatchSize, -1);
         highlighter.setFieldMatcher((name) -> "text".equals(name));
         final Snippet[] snippets =
             highlighter.highlightField("text", query, topDocs.scoreDocs[0].doc, expectedPassages.length);