Add POST /_search/clear_scroll endpoint and deprecate delete scroll endpoint #21510

javanna · 2016-11-11T21:13:31Z

The clear scroll api currently allows to provide a scroll by specifying it either as part of the url (it is effectively the resource that gets deleted) or within the request body. The current api uses the DELETE method though, and we have decided to remove support for providing the request body with any DELETE endpoint in the future. In order to get to this for the next major version, we introduce the new endpoint POST /_search/clear_scroll which replaces the current clear_scroll api and uses POST instead of DELETE. It allows to provide the scroll_id as a url parameter, which is though deprecated (will output a deprecation warning when used) in favour of providing it as part of the request body.

The DELETE /_search/scroll/ is deprecated, hence it will output a deprecation warning whenever used. The DELETE endpoints will be removed in 6.0, as well as the support for providing the scroll_id as a url parameter against the POST endpoint.

Relates to #8217
Relates to #21453

javanna · 2016-11-11T21:15:36Z

rest-api-spec/src/main/resources/rest-api-spec/test/scroll/10_basic.yaml

@@ -62,7 +62,7 @@
  - length: {hits.hits:      0    }

  - do:
-      clear_scroll:
+      delete_scroll:


this is pretty much the breaking change in terms of api. the new clear_scroll doesn't support the scroll_id parameter, so these calls need to be migrated to delete_scroll, or the scroll needs to be provided as part of the request body. Not sure what kinds of problems this can cause in the clients, maybe somebody has suggestions

polyfractal · 2016-11-14T14:01:51Z

Technically, this break will work fine for the clients in regards to testing. E.g. if the client doesn't support the new delete_scroll, it'll just fail the test (as it should).

But for the end-user, this is gonna be problematic. If an end-user is currently using clear_scroll, they provide the scroll ID as part of the endpoint params, e.g:

$client->clear_scroll([
  'scrollId' => 'abc123'
]);

But this change will make that illegal, so they have to adjust their code to move it into the body:

$client->clear_scroll([
  'body' => [
    'scrollId' => 'abc123'
  ]
]);

Or change code to use the new delete_scroll() endpoint. Either way, it's a code-change in a minor release.

So any of the clients following SemVer are going to be in a sticky place, since this is a pretty noticeable API breaking change for a 5.x release. They'd really have to bump the major version to allow this, or start doing magic under the covers to insert the scroll ID into the body (which isn't currently done anywhere, to my knowledge).

To summarize:

clear_scroll -> delete_scroll is fine
Adding back a POST clear_scroll is problematic, since client's can't leave the old DELETE clear_scroll laying around as a deprecated (but usable) endpoint. Ideally, the new POST version would be named something entirely different so it's just adding functionality, not breaking.

javanna · 2016-11-14T14:19:02Z

thanks a lot for your explanation @polyfractal ! I have no intentions to make a breaking change for the clients with this PR, I was just not sure it would break anything. The rename was ideal, as calling the new endpoint delete sounded funny because it uses POST. At this point I would leave clear_scroll as-is with the delete method and come up with a different name for the POST endpoint. Makes sense?

polyfractal · 2016-11-14T21:56:40Z

++ that'd work. :)

I'm afraid I don't have any ideas about an alternative name though :(

javanna · 2016-11-21T15:22:34Z

I made the change fully bw compatible now, @polyfractal mind having another look? Maybe @clintongormley wants to look too?

polyfractal · 2016-11-21T17:01:16Z

rest-api-spec/src/main/resources/rest-api-spec/api/clear_scroll_post.json

@@ -0,0 +1,16 @@
+{
+  "clear_scroll_post": {


I wonder if this should be post_clear_scroll, just to follow the http-verb-preceding-noun convention (put_template, etc)? TBH my level of caring is like 10% though :) @elastic/es-clients any thoughts?

makes sense to me

polyfractal · 2016-11-21T17:02:18Z

Left a minor comment, and I don't particularly care which way it goes.

REST stuff LGTM! Thanks for making this client friendly :)

clintongormley · 2016-11-22T12:09:21Z

What about doing the following instead:

POST _search/clear_scroll accepts a body with the scroll ID
POST _search/clear_scroll also accepts the URL and query string param form of scroll_id (but add them as deprecated)
change the clear_scroll REST spec to use POST _search/clear_scroll
deprecate use of DELETE _search/scroll

This way the clients will continue to work as before, and users will get the deprecated warnings if they try to pass the scroll ID in the URL/query string.

javanna · 2016-11-22T19:34:24Z

@clintongormley did you mean to deprecate DELETE _search/scroll completely? That would mean that the only correct way to provide a scroll_id would be via POST and by providing it as part of the request body? I thought that we wanted to leave the DELETE _search/scroll/${scroll_id} endpoint around. Also having a new clean api that only accepts the body for POST, that could be a better trade-off than having deprecation warnings in there too? What do you think?

clintongormley · 2016-11-23T09:12:34Z

did you mean to deprecate DELETE _search/scroll completely?

Yes. Why have two ways to do it?

That would mean that the only correct way to provide a scroll_id would be via POST and by providing it as part of the request body?

Yes, it's the only reliable way to do it. For the URL and QS versions, it fails when your scroll IDs are too long.

Also having a new clean api that only accepts the body for POST, that could be a better trade-off than having deprecation warnings in there too? What do you think?

This is to handle the change in the client. Users will keep using the clear_scroll method, but internally it'll use the POST endpoint. But they may well be calling it as clear_scroll( scroll_id => 'foo') when they should move to passing it in the body.

So we don't break anything for the user in 5.x, but instead warn them that this will break in 6.0

…ndpoint The clear scroll api currently allows to provide a scroll by specifying it either as part of the url (it is effectively the resource that gets deleted) or within the request body. The current api uses the DELETE method though, and we have decided to remove support for providing the request body with any DELETE endpoint in the future. In order to get to this for the next major version, we introduce the new endpoint `POST /_search/clear_scroll` which replaces the current clear_scroll api and uses POST instead of DELETE. It allows to provide the `scroll_id` as a url parameter, which is though deprecated (will output a deprecation warning when used) in favour of providing it as part of the request body. The `DELETE /_search/scroll/` is deprecated, hence it will output a deprecation warning whenever used. The DELETE endpoints will be removed in 6.0, as well as the support for providing the scroll_id as a url parameter against the POST endpoint. Relates to elastic#8217 Relates to elastic#21453

javanna · 2016-11-23T10:51:12Z

thanks @clintongormley I have updated content, title and description of the PR.

javanna · 2016-11-23T10:52:36Z

docs/reference/search/request/scroll.asciidoc

+{
+    "scroll_id" : [
+      "_all"
+    ]


this one looks a bit odd, looked better providing _all as part of the url, but maybe that's ok

Perhaps this could be replaced with supporting * as a scroll ID

Actually we probably don't support wildcard matching, so using _all is better

clintongormley · 2016-11-23T12:03:06Z

core/src/main/java/org/elasticsearch/rest/action/search/RestClearScrollAction.java

+        }
+        if (request.method() == POST && request.hasParam("scroll_id")) {
+            deprecationLogger.deprecated("Deprecated [POST /_search/clear_scroll/{scroll_id}] endpoint used, " +
+                    "expected [POST /_search/clear_scroll] instead. The scroll_id should rather be provided as part of the request body");


should rather -> must

clintongormley · 2016-11-23T12:07:32Z

docs/reference/search/request/scroll.asciidoc

@@ -165,23 +165,24 @@ DELETE /_search/scroll
 // CONSOLE
 // TEST[catch:missing]

-All search contexts can be cleared with the `_all` parameter:
+The recommended way to provide the `scroll_id` is by placing it in the request


Possibly replace this block with:

deprecated[5.2.0,Providing the scroll ID as part of the URL or as a query string parameter has been deprecated in favour of specifying it in the request body. The `DELETE _search/scroll` end points are deprecated in favour of `POST _search/clear_scroll]

clintongormley · 2016-11-23T12:08:44Z

rest-api-spec/src/main/resources/rest-api-spec/api/clear_scroll.json

@@ -14,7 +14,7 @@
      "params": {}
    },
    "body": {
-      "description": "A comma-separated list of scroll IDs to clear if none was specified via the scroll_id parameter"
+      "description": "A comma-separated list of scroll IDs to clear, has the precedence over the scroll_id parameter"


has precedence

clintongormley · 2016-11-23T12:09:40Z

rest-api-spec/src/main/resources/rest-api-spec/api/delete_scroll.json

@@ -0,0 +1,20 @@
+{


Why add this at all? This will result in adding a new method to the clients.

Is it just there for bwc testing?

no you are right. it is a leftover, due to the fact that I had thought POST shouldn't support the query_string parameter at all, now that it does, I can get right of this new api.

oh wait though, DELETE is done against _search/scroll while POST is done against _search/clear_scroll . I don't think we can express that in our spec without having two apis. POST _search/scroll is already taken by the search api.

javanna · 2017-01-30T17:13:50Z

quick update: this depends on #21888 otherwise it becomes very hard to properly test the change.

javanna · 2017-08-03T06:50:20Z

why was this closed @martijnvg ?

cbuescher · 2018-03-13T11:49:16Z

@javanna is this PR still something we want to do?
//cc @elastic/es-search-aggs

javanna · 2018-03-13T12:06:15Z

yes

colings86 · 2018-10-24T10:25:45Z

@javanna since #21888 is now resolved is this able to progress now (should we remove the stalled label)?

javanna · 2018-10-24T14:23:49Z

yes I was just thinking about this recently. I guess I will need to open a new PR at this point, I need to find time to get back to this.

javanna added :Scroll >deprecation >enhancement v5.1.1 labels Nov 11, 2016

javanna commented Nov 11, 2016

View reviewed changes

javanna force-pushed the enhancement/clear_scroll_post branch 2 times, most recently from 7b12249 to b6e8c4b Compare November 21, 2016 15:20

$polyfractal$

polyfractal reviewed Nov 21, 2016

View reviewed changes

javanna force-pushed the enhancement/clear_scroll_post branch from b6e8c4b to 50403ed Compare November 23, 2016 10:49

javanna changed the title ~~Add POST /_search/clear_scroll endpoint and deprecate delete scroll with request body~~ Add POST /_search/clear_scroll endpoint and deprecate delete scroll endpoint Nov 23, 2016

javanna added v5.2.0 and removed v5.1.1 labels Nov 23, 2016

javanna commented Nov 23, 2016

View reviewed changes

clarified spec

33b0491

clintongormley reviewed Nov 23, 2016

View reviewed changes

addressed review comments

b04cef5

clintongormley added v5.3.0 and removed v5.2.0 labels Jan 24, 2017

javanna added the stalled label Jan 30, 2017

clintongormley added v5.4.0 and removed v5.3.0 labels Feb 7, 2017

clintongormley added v5.4.1 and removed v5.4.0 labels Apr 28, 2017

clintongormley added v5.4.2 and removed v5.4.1 labels May 15, 2017

javanna mentioned this pull request Jun 9, 2017

Reject DELETE requests with a body #21453

Closed

clintongormley added v5.4.3 and removed v5.4.2 labels Jun 14, 2017

clintongormley added v5.4.4 and removed v5.4.3 labels Jun 27, 2017

martijnvg closed this Aug 2, 2017

javanna changed the base branch from 5.x to 5.6 August 3, 2017 06:57

javanna reopened this Aug 3, 2017

clintongormley added :Search/Search Search-related issues that do not fall into other categories and removed :Scroll labels Feb 14, 2018

nik9000 mentioned this pull request Mar 22, 2018

Low Level REST Client: Support host selection #29211

Closed

rjernst removed the review label Oct 10, 2018

javanna removed stalled v5.4.4 labels Oct 24, 2018

javanna closed this Oct 24, 2018

Add POST /_search/clear_scroll endpoint and deprecate delete scroll endpoint #21510

Add POST /_search/clear_scroll endpoint and deprecate delete scroll endpoint #21510

Uh oh!

Conversation

javanna commented Nov 11, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

polyfractal commented Nov 14, 2016

Uh oh!

javanna commented Nov 14, 2016

Uh oh!

polyfractal commented Nov 14, 2016

Uh oh!

javanna commented Nov 21, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

polyfractal commented Nov 21, 2016

Uh oh!

clintongormley commented Nov 22, 2016

Uh oh!

javanna commented Nov 22, 2016

Uh oh!

clintongormley commented Nov 23, 2016

Uh oh!

javanna commented Nov 23, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

javanna Nov 23, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

javanna Nov 23, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

javanna commented Jan 30, 2017

Uh oh!

javanna commented Aug 3, 2017

Uh oh!

cbuescher commented Mar 13, 2018

Uh oh!

javanna commented Mar 13, 2018

Uh oh!

colings86 commented Oct 24, 2018

Uh oh!

javanna commented Oct 24, 2018

Uh oh!

Uh oh!

javanna commented Nov 11, 2016 •

edited

Loading

javanna Nov 23, 2016 •

edited

Loading

javanna Nov 23, 2016 •

edited

Loading