Fixed naming inconsistency for fields/stored_fields in the APIs #20166

jimczi · 2016-08-25T15:18:13Z

This change replaces the fields parameter with stored_fields when it makes sense.
This is dictated by the renaming we made in #18943 for the search API.

The following list of endpoint has been changed to use stored_fields instead of fields:

get
mget
explain

The documentation and the rest API spec has been updated to cope with the changes for the following APIs:

delete_by_query
get
mget
explain

The fields parameter has been deprecated for the following APIs:

update
bulk

These APIs now support _source as a parameter to filter the _source of the updated document to be returned.

Some APIs still have the fields parameter for various reasons:

cat.fielddata: the fields paramaters relates to the fielddata fields that should be printed.
indices.clear_cache: used to indicate which fielddata fields should be cleared.
indices.get_field_mapping: used to filter fields in the mapping.
indices.stats: get stats on fields (stored or not stored).
termvectors: fields are retrieved from the stored fields if possible and extracted from the _source otherwise.
mtermvectors:
nodes.stats: the fields parameter is used to concatenate completion_fields and fielddata_fields so it's not related to stored_fields at all.

Fixes #20155

jpountz · 2016-08-26T07:53:30Z

The change looks good and your explanation about what APIs you modified makes sense to me. Should we have nicer backward compatibility and use ParseField to make fields a deprecated alias of stored_fields like we did for docvalue_fields and fielddata_fields? (See also this comment: #18943 (comment))

jimczi · 2016-08-26T08:13:49Z

Should we have nicer backward compatibility and use ParseField to make fields a deprecated alias of stored_fields like we did for docvalue_fields and fielddata_fields? (See also this comment: #18943 (comment))

Sure or maybe we can just throw a nice exception if fields is used ? It would be consistent with what we have for the Search API:

elasticsearch/core/src/main/java/org/elasticsearch/search/builder/SearchSourceBuilder.java

Line 1070 in 50b47aa

throw new ParsingException(parser.getTokenLocation(), "The field [" +

jpountz · 2016-08-26T09:00:53Z

An exception would work for me even though when it comes to the DSL, I like having things deprecated better: unlike Java there is no compile time error, it only fails at runtime. Moreover, major version upgrades are already challenging on their own so if we can avoid breaking changes in the DSL, I think that's better.

clintongormley · 2016-08-26T10:51:56Z

I like having things deprecated better

Given that this is not just a change in parameter name but also a change in behaviour, I'd prefer an exception over a deprecation. It is easier to rename the parameter in your code than to figure out why you're getting bad results.

Perhaps we could add deprecation logging for fields in 2.4.1?

clintongormley · 2016-08-26T10:55:16Z

In bulk and update, I'd deprecate the use of fields and add support for _source instead. This makes it more consistent.

The termvectors and mtermvectors APIs are trickier... Perhaps leave them as they are for now.

jimczi · 2016-08-29T07:20:28Z

I think it's ok to add _source in the update request but it conflicts with the java API where we use setSource and source to create the request from a bytes array . Since it's not used in the rest API and that source in this case has nothing to do with the source of the document, is it ok to rename setSource to something like fromBytes ?

jpountz · 2016-08-29T12:26:47Z

Since it's not used in the rest API and that source in this case has nothing to do with the source of the document, is it ok to rename setSource to something like fromBytes ?

Not sure about the name, but definitely +1 on having the Java API as close as possible to the REST API and then renaming internal Java APIs if there is conflict. Back to the name... maybe fromXContent since we expect these bytes to be some form of xcontent and the name would be consistent with what we use in query builders to parse incoming bytes (even though the bytes are provided differently).

jimczi · 2016-08-30T14:32:59Z

I pushed a change that deprecate the fields parameter in the bulk/update API and adds the support for _source filtering.
@jpountz could you please take another look ?

This change replaces the fields parameter with stored_fields when it makes sense. This is dictated by the renaming we made in #18943 for the search API. The following list of endpoint has been changed to use `stored_fields` instead of `fields`: * get * mget * explain The documentation and the rest API spec has been updated to cope with the changes for the following APIs: * delete_by_query * get * mget * explain Some APIs still have the `fields` parameter for various reasons: * update: the fields are extracted from the _source directly. * bulk: the fields parameter is used but fields are extracted from the source directly so it is allowed to have non-stored fields. * cat.fielddata: the fields paramaters relates to the fielddata fields that should be printed. * indices.clear_cache: used to indicate which fielddata fields should be cleared. * indices.get_field_mapping: used to filter fields in the mapping. * indices.stats: get stats on fields (stored or not stored). * termvectors: fields are retrieved from the stored fields if possible and extracted from the _source otherwise. * mtermvectors: * nodes.stats: the fields parameter is used to concatenate completion_fields and fielddata_fields so it's not related to stored_fields at all. Fixes #20155

… filtering. fixes

…n request

markharwood · 2016-09-13T09:39:56Z

docs/reference/docs/update.asciidoc

        "lang": "painless",
        "params" : {
-            "tag" : "blue"
+            "tag" : "green"
        }


^ tag:green inconsistent with the example description which talks about tag:blue

I changed the description to mention the green tag, I had to change the logic since we use the same doc for all CONSOLE snippet and it will break the next snippet if the document is removed.

jimczi · 2016-09-13T12:23:38Z

Thanks @markharwood I pushed a commit to address your comment.

nik9000 · 2016-09-13T13:33:16Z

docs/groovy-api/search.asciidoc

@@ -72,7 +72,7 @@ def search = node.client.search {
    source {
        query {
            query_string(
-                fields: ["test"],
+                stored_fields: ["test"],


I think this change is incorrect. query_string still takes fields.

nik9000 · 2016-09-13T14:15:02Z

core/src/main/java/org/elasticsearch/index/termvectors/TermVectorsService.java

-                index.addField(field, text.toString(), analyzer);
+            if (entry.getValue() instanceof List) {
+                for (Object text : entry.getValue()) {
+                    index.addField(field, text.toString(), analyzer);


Some mean person is going to write "fields": ["bleh", null, "blah"] and complain about the weird exception at this line. Maybe just throw an IllegalArgumentException here.

If it is null.

This list is the response of a get request so we should never have null values.

Ah! All good then.

nik9000 · 2016-09-13T14:28:17Z

I left a bunch of minor stuff. Asked for more docs, more formatting, and modernizing a few tests. I didn't see anything major. I think it is the right thing to do and we should get it in sooner rather than later so we can get it in 5.0's next release. My instinct is that we're going to find some subtle bug that I should have caught on review and I'm going to facepalm but it really does look good to me other than all the minor stuff I left.

jimczi · 2016-09-13T15:59:00Z

Thank you so much for the review @nik9000 ! I pushed a commit to hopefully address all your feedbacks. Can you take another look ?

nik9000 · 2016-09-13T16:05:27Z

docs/reference/docs/get.asciidoc

+--------------------------------------------------
+GET twitter/tweet/1?routing=user1,stored_fields=tags,counter
+--------------------------------------------------
+


This one is missing // CONSOLE I think.

nik9000 · 2016-09-13T16:07:05Z

Left one minor thing on the last commit. I'm going to take a quick break and reread the whole thing one last time after that but I expect it'll be fine.

nik9000 · 2016-09-13T16:39:53Z

core/src/main/java/org/elasticsearch/action/update/UpdateHelper.java

@@ -298,8 +304,24 @@ public GetResult extractGetResult(final UpdateRequest request, String concreteIn
            }
        }

+        BytesReference sourceFilteredAsBytes = sourceAsBytes;
+        if (request.fetchSource() != null) {


Is it worth optimizing the case where fetchSource is true and includes and excludes are empty and the xcontents line up? Then you can just return the bytes you have like we do for search lookups.

What about the case when request.fetchSource().fetchSource() is false? Why do we even have a boolean there? Maybe we should just use null instead? It doesn't make sense to have a FetchSourceContext with fetchSource = false and includes = ["something"].

Maybe just handle the boolean for this PR, but we should think about removing it....

nik9000 · 2016-09-13T16:48:07Z

LGTM. I left some very minor stuff but you can merge as is if you feel the need and/or fix what I left and merge without another review.

… Fixed docs

This change replaces the fields parameter with stored_fields when it makes sense. This is dictated by the renaming we made in #18943 for the search API. The following list of endpoint has been changed to use `stored_fields` instead of `fields`: * get * mget * explain The documentation and the rest API spec has been updated to cope with the changes for the following APIs: * delete_by_query * get * mget * explain The `fields` parameter has been deprecated for the following APIs (it is replaced by _source filtering): * update: the fields are extracted from the _source directly. * bulk: the fields parameter is used but fields are extracted from the source directly so it is allowed to have non-stored fields. Some APIs still have the `fields` parameter for various reasons: * cat.fielddata: the fields paramaters relates to the fielddata fields that should be printed. * indices.clear_cache: used to indicate which fielddata fields should be cleared. * indices.get_field_mapping: used to filter fields in the mapping. * indices.stats: get stats on fields (stored or not stored). * termvectors: fields are retrieved from the stored fields if possible and extracted from the _source otherwise. * mtermvectors: * nodes.stats: the fields parameter is used to concatenate completion_fields and fielddata_fields so it's not related to stored_fields at all. Fixes #20155

jimczi · 2016-09-13T19:19:14Z

Thanks again @nik9000 ! It is merged in 5.0 and 5.x.

jimczi added >bug >breaking :Distributed Indexing/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. v5.0.0-beta1 labels Aug 25, 2016

jimczi mentioned this pull request Aug 29, 2016

Fix docs that uses += to add an element in a list even though painless does not accept it. #20213

Merged

jimczi added the review label Sep 8, 2016

jimczi added 3 commits September 13, 2016 11:09

Deprecate fields in the update/bulk API and add support for _source…

8272c2a

… filtering. fixes

Throw exception when fields parameter is used on get, mget and explai…

3733951

…n request

jimczi added v6.0.0-alpha1 v5.1.1 labels Sep 13, 2016

markharwood reviewed Sep 13, 2016
View reviewed changes

Fix docs and rest api spec

80788c3

nik9000 reviewed Sep 13, 2016
View reviewed changes

Fix typos, docs and tests

7f0ac15

nik9000 reviewed Sep 13, 2016
View reviewed changes

Add missing CONSOLE and fix snippets

da262c3

nik9000 reviewed Sep 13, 2016
View reviewed changes

Filter _source only if includes or excludes contain fields to filter.…

905aabd

… Fixed docs

jimczi merged commit 1764ec5 into elastic:master Sep 13, 2016

jimczi deleted the fields_renaming branch September 13, 2016 18:54

clintongormley removed the v5.1.1 label Dec 8, 2016

joschi mentioned this pull request Jul 31, 2017

API Browser Broken after Upgrade to 2.3 (stored_fields) Graylog2/graylog2-server#4042

Closed

Fixed naming inconsistency for fields/stored_fields in the APIs #20166

Fixed naming inconsistency for fields/stored_fields in the APIs #20166

Uh oh!

Conversation

jimczi commented Aug 25, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jpountz commented Aug 26, 2016

Uh oh!

jimczi commented Aug 26, 2016

Uh oh!

jpountz commented Aug 26, 2016

Uh oh!

clintongormley commented Aug 26, 2016

Uh oh!

clintongormley commented Aug 26, 2016

Uh oh!

jimczi commented Aug 29, 2016

Uh oh!

jpountz commented Aug 29, 2016

Uh oh!

jimczi commented Aug 30, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimczi commented Sep 13, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nik9000 commented Sep 13, 2016

Uh oh!

jimczi commented Sep 13, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nik9000 commented Sep 13, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nik9000 commented Sep 13, 2016

Uh oh!

jimczi commented Sep 13, 2016

Uh oh!

Uh oh!

jimczi commented Aug 25, 2016 •

edited

Loading