Add support for field aliases. (#32172)

* Add basic support for field aliases in index mappings. (#31287)
* Allow for aliases when fetching stored fields. (#31411)
* Add tests around accessing field aliases in scripts. (#31417)
* Add documentation around field aliases. (#31538)
* Add validation for field alias mappings. (#31518)
* Return both concrete fields and aliases in DocumentFieldMappers#getMapper. (#31671)
* Make sure that field-level security is enforced when using field aliases. (#31807)
* Add more comprehensive tests for field aliases in queries + aggregations. (#31565)
* Remove the deprecated method DocumentFieldMappers#getFieldMapper. (#32148)

Author: jtibshirani

docs/reference/indices/clearcache.asciidoc

@@ -16,7 +16,8 @@ explicitly by setting `query`, `fielddata` or `request`.
 
 All caches relating to a specific field(s) can also be cleared by
 specifying `fields` parameter with a comma delimited list of the
-relevant fields.
+relevant fields. Note that the provided names must refer to concrete
+fields -- objects and field aliases are not supported.
 
 [float]
 === Multi Index

docs/reference/mapping.asciidoc

@@ -124,8 +124,10 @@ fields to an existing index with the <<indices-put-mapping,PUT mapping API>>.
 
 Other than where documented, *existing field mappings cannot be
 updated*. Changing the mapping would mean invalidating already indexed
-documents.  Instead, you should create a new index with the correct mappings
-and <<docs-reindex,reindex>> your data into that index.
+documents. Instead, you should create a new index with the correct mappings
+and <<docs-reindex,reindex>> your data into that index. If you only wish
+to rename a field and not change its mappings, it may make sense to introduce
+an <<alias, `alias`>> field.
 
 [float]
 == Example mapping

docs/reference/mapping/types.asciidoc

@@ -40,6 +40,8 @@ string::        <<text,`text`>> and <<keyword,`keyword`>>
 
 <<parent-join>>::   Defines parent/child relation for documents within the same index
 
+<<alias>>::         Defines an alias to an existing field.
+
 <<feature>>::       Record numeric features to boost hits at query time.
 
 <<feature-vector>>:: Record numeric feature vectors to boost hits at query time.
@@ -58,6 +60,8 @@ the <<analysis-standard-analyzer,`standard` analyzer>>, the
 This is the purpose of _multi-fields_.  Most datatypes support multi-fields
 via the <<multi-fields>> parameter.
 
+include::types/alias.asciidoc[]
+
 include::types/array.asciidoc[]
 
 include::types/binary.asciidoc[]

docs/reference/mapping/types/alias.asciidoc

@@ -0,0 +1,101 @@
+[[alias]]
+=== Alias datatype
+
+An `alias` mapping defines an alternate name for a field in the  index.
+The alias can be used in place of the target field in <<search, search>> requests,
+and selected other APIs like <<search-field-caps, field capabilities>>.
+
+[source,js]
+--------------------------------
+PUT trips
+{
+  "mappings": {
+    "_doc": {
+      "properties": {
+        "distance": {
+          "type": "long"
+        },
+        "route_length_miles": {
+          "type": "alias",
+          "path": "distance" // <1>
+        },
+        "transit_mode": {
+          "type": "keyword"
+        }
+      }
+    }
+  }
+}
+
+GET _search
+{
+  "query": {
+    "range" : {
+      "route_length_miles" : {
+        "gte" : 39
+      }
+    }
+  }
+}
+--------------------------------
+// CONSOLE
+
+<1> The path to the target field. Note that this must be the full path, including any parent
+objects (e.g. `object1.object2.field`).
+
+Almost all components of the search request accept field aliases. In particular, aliases can be
+used in queries, aggregations, and sort fields, as well as when requesting `docvalue_fields`,
+`stored_fields`, suggestions, and highlights. Scripts also support aliases when accessing
+field values. Please see the section on <<unsupported-apis, unsupported APIs>> for exceptions.
+
+In some parts of the search request and when requesting field capabilities, field wildcard patterns can be
+provided. In these cases, the wildcard pattern will match field aliases in addition to concrete fields:
+
+[source,js]
+--------------------------------
+GET trips/_field_caps?fields=route_*,transit_mode
+--------------------------------
+// CONSOLE
+// TEST[continued]
+
+[[alias-targets]]
+==== Alias targets
+
+There are a few restrictions on the target of an alias:
+
+  * The target must be a concrete field, and not an object or another field alias.
+  * The target field must exist at the time the alias is created.
+  * If nested objects are defined, a field alias must have the same nested scope as its target.
+
+Additionally, a field alias can only have one target. This means that it is not possible to use a
+field alias to query over multiple target fields in a single clause.
+
+[[unsupported-apis]]
+==== Unsupported APIs
+
+Writes to field aliases are not supported: attempting to use an alias in an index or update request
+will result in a failure. Likewise, aliases cannot be used as the target of `copy_to`.
+
+Because alias names are not present in the document source, aliases cannot be used when performing
+source filtering. For example, the following request will return an empty result for `_source`:
+
+[source,js]
+--------------------------------
+GET /_search
+{
+  "query" : {
+    "match_all": {}
+  },
+  "_source": "route_length_miles"
+}
+--------------------------------
+// CONSOLE
+// TEST[continued]
+
+Currently only the search and field capabilities APIs will accept and resolve field aliases.
+Other APIs that accept field names, such as <<docs-termvectors, term vectors>>, cannot be used
+with field aliases.
+
+Finally, some queries, such as `terms`, `geo_shape`, and `more_like_this`, allow for fetching query
+information from an indexed document. Because field aliases aren't supported when fetching documents,
+the part of the query that specifies the lookup path cannot refer to a field by its alias.

modules/lang-expression/src/test/java/org/elasticsearch/script/expression/ExpressionTests.java

@@ -20,27 +20,52 @@
 package org.elasticsearch.script.expression;
 
 import org.elasticsearch.common.settings.Settings;
-import org.elasticsearch.index.IndexService;
-import org.elasticsearch.index.query.QueryShardContext;
+import org.elasticsearch.index.fielddata.AtomicNumericFieldData;
+import org.elasticsearch.index.fielddata.IndexNumericFieldData;
+import org.elasticsearch.index.fielddata.SortedNumericDoubleValues;
+import org.elasticsearch.index.mapper.MapperService;
+import org.elasticsearch.index.mapper.NumberFieldMapper.NumberFieldType;
+import org.elasticsearch.index.mapper.NumberFieldMapper.NumberType;
 import org.elasticsearch.script.ScriptException;
 import org.elasticsearch.script.SearchScript;
 import org.elasticsearch.search.lookup.SearchLookup;
-import org.elasticsearch.test.ESSingleNodeTestCase;
+import org.elasticsearch.test.ESTestCase;
 
+import java.io.IOException;
 import java.text.ParseException;
 import java.util.Collections;
 
-public class ExpressionTests extends ESSingleNodeTestCase {
-    ExpressionScriptEngine service;
-    SearchLookup lookup;
+import static org.mockito.Matchers.anyInt;
+import static org.mockito.Matchers.anyObject;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class ExpressionTests extends ESTestCase {
+    private ExpressionScriptEngine service;
+    private SearchLookup lookup;
 
     @Override
     public void setUp() throws Exception {
         super.setUp();
-        IndexService index = createIndex("test", Settings.EMPTY, "type", "d", "type=double");
+
+        NumberFieldType fieldType = new NumberFieldType(NumberType.DOUBLE);
+        MapperService mapperService = mock(MapperService.class);
+        when(mapperService.fullName("field")).thenReturn(fieldType);
+        when(mapperService.fullName("alias")).thenReturn(fieldType);
+
+        SortedNumericDoubleValues doubleValues = mock(SortedNumericDoubleValues.class);
+        when(doubleValues.advanceExact(anyInt())).thenReturn(true);
+        when(doubleValues.nextValue()).thenReturn(2.718);
+
+        AtomicNumericFieldData atomicFieldData = mock(AtomicNumericFieldData.class);
+        when(atomicFieldData.getDoubleValues()).thenReturn(doubleValues);
+
+        IndexNumericFieldData fieldData = mock(IndexNumericFieldData.class);
+        when(fieldData.getFieldName()).thenReturn("field");
+        when(fieldData.load(anyObject())).thenReturn(atomicFieldData);
+
         service = new ExpressionScriptEngine(Settings.EMPTY);
-        QueryShardContext shardContext = index.newQueryShardContext(0, null, () -> 0, null);
-        lookup = new SearchLookup(index.mapperService(), shardContext::getForField, null);
+        lookup = new SearchLookup(mapperService, ignored -> fieldData, null);
     }
 
     private SearchScript.LeafFactory compile(String expression) {
@@ -50,22 +75,38 @@ private SearchScript.LeafFactory compile(String expression) {
 
     public void testNeedsScores() {
         assertFalse(compile("1.2").needs_score());
-        assertFalse(compile("doc['d'].value").needs_score());
+        assertFalse(compile("doc['field'].value").needs_score());
         assertTrue(compile("1/_score").needs_score());
-        assertTrue(compile("doc['d'].value * _score").needs_score());
+        assertTrue(compile("doc['field'].value * _score").needs_score());
     }
 
     public void testCompileError() {
         ScriptException e = expectThrows(ScriptException.class, () -> {
-            compile("doc['d'].value * *@#)(@$*@#$ + 4");
+            compile("doc['field'].value * *@#)(@$*@#$ + 4");
         });
         assertTrue(e.getCause() instanceof ParseException);
     }
 
     public void testLinkError() {
         ScriptException e = expectThrows(ScriptException.class, () -> {
-            compile("doc['e'].value * 5");
+            compile("doc['nonexistent'].value * 5");
         });
         assertTrue(e.getCause() instanceof ParseException);
     }
+
+    public void testFieldAccess() throws IOException {
+        SearchScript script = compile("doc['field'].value").newInstance(null);
+        script.setDocument(1);
+
+        double result = script.runAsDouble();
+        assertEquals(2.718, result, 0.0);
+    }
+
+    public void testFieldAccessWithFieldAlias() throws IOException {
+        SearchScript script = compile("doc['alias'].value").newInstance(null);
+        script.setDocument(1);
+
+        double result = script.runAsDouble();
+        assertEquals(2.718, result, 0.0);
+    }
 }

modules/percolator/src/main/java/org/elasticsearch/percolator/PercolateQuery.java

@@ -194,6 +194,10 @@ Query getCandidateMatchesQuery() {
         return candidateMatchesQuery;
     }
 
+    Query getVerifiedMatchesQuery() {
+        return verifiedMatchesQuery;
+    }
+
     // Comparing identity here to avoid being cached
     // Note that in theory if the same instance gets used multiple times it could still get cached,
     // however since we create a new query instance each time we this query this shouldn't happen and thus

modules/percolator/src/main/java/org/elasticsearch/percolator/PercolateQueryBuilder.java

@@ -618,13 +618,13 @@ protected Analyzer getWrappedAnalyzer(String fieldName) {
             docSearcher.setQueryCache(null);
         }
 
-        PercolatorFieldMapper percolatorFieldMapper = (PercolatorFieldMapper) docMapper.mappers().getMapper(field);
-        boolean mapUnmappedFieldsAsString = percolatorFieldMapper.isMapUnmappedFieldAsText();
+        PercolatorFieldMapper.FieldType pft = (PercolatorFieldMapper.FieldType) fieldType;
+        String name = this.name != null ? this.name : pft.name();
         QueryShardContext percolateShardContext = wrap(context);
+        PercolateQuery.QueryStore queryStore = createStore(pft.queryBuilderField,
+            percolateShardContext,
+            pft.mapUnmappedFieldsAsText);
 
-        String name = this.name != null ? this.name : field;
-        PercolatorFieldMapper.FieldType pft = (PercolatorFieldMapper.FieldType) fieldType;
-        PercolateQuery.QueryStore queryStore = createStore(pft.queryBuilderField, percolateShardContext, mapUnmappedFieldsAsString);
         return pft.percolateQuery(name, queryStore, documents, docSearcher, context.indexVersionCreated());
     }
 

modules/percolator/src/main/java/org/elasticsearch/percolator/PercolatorFieldMapper.java

@@ -136,13 +136,19 @@ public PercolatorFieldMapper build(BuilderContext context) {
             fieldType.rangeField = rangeFieldMapper.fieldType();
             NumberFieldMapper minimumShouldMatchFieldMapper = createMinimumShouldMatchField(context);
             fieldType.minimumShouldMatchField = minimumShouldMatchFieldMapper.fieldType();
+            fieldType.mapUnmappedFieldsAsText = getMapUnmappedFieldAsText(context.indexSettings());
+
             context.path().remove();
             setupFieldType(context);
             return new PercolatorFieldMapper(name(), fieldType, defaultFieldType, context.indexSettings(),
                     multiFieldsBuilder.build(this, context), copyTo, queryShardContext, extractedTermsField,
                     extractionResultField, queryBuilderField, rangeFieldMapper, minimumShouldMatchFieldMapper);
         }
 
+        private static boolean getMapUnmappedFieldAsText(Settings indexSettings) {
+            return INDEX_MAP_UNMAPPED_FIELDS_AS_TEXT_SETTING.get(indexSettings);
+        }
+
         static KeywordFieldMapper createExtractQueryFieldBuilder(String name, BuilderContext context) {
             KeywordFieldMapper.Builder queryMetaDataFieldBuilder = new KeywordFieldMapper.Builder(name);
             queryMetaDataFieldBuilder.docValues(false);
@@ -195,6 +201,7 @@ static class FieldType extends MappedFieldType {
         MappedFieldType minimumShouldMatchField;
 
         RangeFieldMapper.RangeFieldType rangeField;
+        boolean mapUnmappedFieldsAsText;
 
         FieldType() {
             setIndexOptions(IndexOptions.NONE);
@@ -209,6 +216,7 @@ static class FieldType extends MappedFieldType {
             queryBuilderField = ref.queryBuilderField;
             rangeField = ref.rangeField;
             minimumShouldMatchField = ref.minimumShouldMatchField;
+            mapUnmappedFieldsAsText = ref.mapUnmappedFieldsAsText;
         }
 
         @Override
@@ -327,7 +335,6 @@ Tuple<List<BytesRef>, Map<String, List<byte[]>>> extractTermsAndRanges(IndexRead
 
     }
 
-    private final boolean mapUnmappedFieldAsText;
     private final Supplier<QueryShardContext> queryShardContext;
     private KeywordFieldMapper queryTermsField;
     private KeywordFieldMapper extractionResultField;
@@ -348,14 +355,9 @@ Tuple<List<BytesRef>, Map<String, List<byte[]>>> extractTermsAndRanges(IndexRead
         this.extractionResultField = extractionResultField;
         this.queryBuilderField = queryBuilderField;
         this.minimumShouldMatchFieldMapper = minimumShouldMatchFieldMapper;
-        this.mapUnmappedFieldAsText = getMapUnmappedFieldAsText(indexSettings);
         this.rangeFieldMapper = rangeFieldMapper;
     }
 
-    private static boolean getMapUnmappedFieldAsText(Settings indexSettings) {
-        return INDEX_MAP_UNMAPPED_FIELDS_AS_TEXT_SETTING.get(indexSettings);
-    }
-
     @Override
     public FieldMapper updateFieldType(Map<String, MappedFieldType> fullNameToFieldType) {
         PercolatorFieldMapper updated = (PercolatorFieldMapper) super.updateFieldType(fullNameToFieldType);
@@ -402,7 +404,7 @@ public Mapper parse(ParseContext context) throws IOException {
 
         Version indexVersion = context.mapperService().getIndexSettings().getIndexVersionCreated();
         createQueryBuilderField(indexVersion, queryBuilderField, queryBuilder, context);
-        Query query = toQuery(queryShardContext, mapUnmappedFieldAsText, queryBuilder);
+        Query query = toQuery(queryShardContext, isMapUnmappedFieldAsText(), queryBuilder);
         processQuery(query, context);
         return null;
     }
@@ -522,7 +524,7 @@ protected String contentType() {
     }
 
     boolean isMapUnmappedFieldAsText() {
-        return mapUnmappedFieldAsText;
+        return ((FieldType) fieldType).mapUnmappedFieldsAsText;
     }
 
     /**

modules/percolator/src/test/java/org/elasticsearch/percolator/CandidateQueryTests.java

@@ -194,8 +194,7 @@ public void testDuel() throws Exception {
         }
         Collections.sort(intValues);
 
-        MappedFieldType intFieldType = mapperService.documentMapper("type").mappers()
-            .getMapper("int_field").fieldType();
+        MappedFieldType intFieldType = mapperService.fullName("int_field");
 
         List<Supplier<Query>> queryFunctions = new ArrayList<>();
         queryFunctions.add(MatchNoDocsQuery::new);
@@ -327,8 +326,7 @@ public void testDuel2() throws Exception {
         stringValues.add("value2");
         stringValues.add("value3");
 
-        MappedFieldType intFieldType = mapperService.documentMapper("type").mappers()
-            .getMapper("int_field").fieldType();
+        MappedFieldType intFieldType = mapperService.fullName("int_field");
         List<int[]> ranges = new ArrayList<>();
         ranges.add(new int[]{-5, 5});
         ranges.add(new int[]{0, 10});