version set in ingest pipeline #27573

PnPie · 2017-11-28T20:13:23Z

As discussed in #27242, I make this PR firstly for support of setting document version and version type in set processor of an ingest pipeline.
If it's okay, I'm really happy to try to create a new processor later for version set if needed.

Add support for setting document version and version type in set processor of an ingest pipeline.

elasticmachine · 2017-11-28T20:13:25Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

DaveCTurner · 2017-11-28T21:13:05Z

@elasticmachine please test this

PnPie · 2017-11-28T23:10:55Z

I see the test fails because there are some lines longer than 140 characters. So I modified it.

rjernst · 2017-12-21T21:13:52Z

@elasticmachine test this please

elasticmachine · 2017-12-22T19:21:13Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

rjernst · 2017-12-22T19:21:30Z

@elasticmachine test this please

rjernst · 2017-12-22T21:10:05Z

@elasticmachine test this please

PnPie · 2017-12-24T11:18:24Z

Hello,
Seems the build failed cause some problems encountered while installing AnalysisIcuPlugin ? But I didn't reproduced it locally by running build for :docs subproject.

21:36:49 * What went wrong:
21:36:49 Execution failed for task ':docs:integTestCluster#installAnalysisIcuPlugin'.
21:36:49 > A problem occurred starting process 'command '/var/lib/jenkins/workspace/elastic+elasticsearch+pull-request/docs/build/cluster/integTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/bin/elasticsearch-plugin''

rjernst · 2018-01-09T22:42:07Z

@PnPie Sorry I did not see the test failure output over the holidays. I will run them again.

@elasticmachine test this please

rjernst · 2018-01-10T01:23:47Z

@PnPie can you please merge the latest master into your branch? We have started using the gradle wrapper, and CI is expecting the wrapper to exist in order to run tests.

…_pipeline_version_set

PnPie · 2018-01-10T18:52:45Z

Hello @rjernst ,
I just merged master into this.

rjernst · 2018-01-16T05:03:29Z

@PnPie I'm sorry to have to ask again, but there was large move of files from core to server, so this needs another merge with master.

…_pipeline_version_set

PnPie · 2018-01-16T19:52:40Z

@rjernst there is no problem, I merged again. And just feel free to push to this branch if needed :)

rjernst · 2018-01-24T06:56:11Z

@elasticmachine test this please

martijnvg · 2018-01-30T06:10:01Z

@elasticmachine test this please

(previous build results were removed)

martijnvg

Hey @PnPie, I took a look at your PR and left some suggestions. Can you also add a yml test file in ./modules/ingest-common/src/test/resources/rest-api-spec/test/ingest to tests version and version_type for set and append processors?

martijnvg · 2018-01-30T06:11:07Z

modules/ingest-common/src/test/java/org/elasticsearch/ingest/common/AppendProcessorTests.java

-            values.add(value);
-            appendProcessor = createAppendProcessor(randomMetaData.getFieldName(), value);
+        int valuesSize = randomIntBetween(0, 10);
+        if (randomMetaData == IngestDocument.MetaData.VERSION) {


Can version and version_type be tested in a dedicated test? I think that makes these tests easier to read.

martijnvg · 2018-01-30T06:11:27Z

modules/ingest-common/src/test/java/org/elasticsearch/ingest/common/SetProcessorTests.java

@@ -101,10 +102,28 @@ public void testSetExistingNullFieldWithOverrideDisabled() throws Exception {

    public void testSetMetadata() throws Exception {
        IngestDocument.MetaData randomMetaData = randomFrom(IngestDocument.MetaData.values());
-        Processor processor = createSetProcessor(randomMetaData.getFieldName(), "_value", true);
+        Processor processor;


Same as previous comment.

martijnvg · 2018-01-30T06:15:14Z

server/src/main/java/org/elasticsearch/ingest/IngestDocument.java

@@ -73,6 +73,17 @@ public IngestDocument(String index, String type, String id, String routing, Stri
        this.ingestMetadata.put(TIMESTAMP, ZonedDateTime.now(ZoneOffset.UTC));
    }

+    public IngestDocument(String index, String type, String id, String routing,String parent,


Instead of adding a new constructor, can the existing constructor be modified to also accept version and versionType? I know this requires changing more tests, but I think this class should only have two constructors.

…_pipeline_version_set

PnPie · 2018-02-10T22:22:30Z

Hello @martijnvg :)
I did the changes you mentioned but I'm not really familiar with the yml test, so I pushed sth. I did and I may have some questions.
For the version type, seems we can't get it directly from a document ? All we can do is to add a parameter to GET request (ex. version_type=internal) to get documents with specified version_type ? But can we do this in yml test ?

And I also find that specifying version_type in a GET request seems doesn't work. I created an index with a document all simple, but I can GET the document by specifying whatever for version_type, shall we have a bug here ?

PnPie · 2018-02-10T22:27:13Z

modules/ingest-common/src/test/resources/rest-api-spec/test/ingest/170_version.yml

+        index: test
+        type: test
+        id: 1
+  - match: { _version: 3 }


Here for a GET request, can we put parameters to specify version_type for example ?

Yes, version_type and version can be specified below the id parameter.

martijnvg

Thanks for updating the pr @PnPie!

All we can do is to add a parameter to GET request (ex. version_type=internal) to get documents with specified version_type ? But can we do this in yml test ?

This can be specified in yaml tests. For example look at ./elasticsearch/rest-api-spec/src/main/resources/rest-api-spec/test/get/90_versions.yml file for how version_type can be used in yaml tests.

And I also find that specifying version_type in a GET request seems doesn't work. I created an index with a document all simple, but I can GET the document by specifying whatever for version_type, shall we have a bug here ?

I don't think there is a bug here, a version conflict is being thrown if the provided version does not match with the version that a document has. The difference in behavior between version_type is subtle. (most notable is that external version type fails if expected version is unequal to actual version whereas external_gte version_type fails if external version is smaller than the actual version)

After thinking more about version_type and version, I think it does not make sense to specifically test these meta fields with the append processor, because only a single version / version_type can be specified per document. So can you remove the tests you added for append processor?

martijnvg · 2018-02-13T07:02:10Z

modules/ingest-common/src/test/resources/rest-api-spec/test/ingest/170_version.yml

+        index: test
+        type: test
+        id: 1
+  - match: { _version: 3 }


Yes, version_type and version can be specified below the id parameter.

martijnvg · 2018-02-13T07:49:31Z

modules/ingest-common/src/test/resources/rest-api-spec/test/ingest/170_version.yml

+              {
+                "append" : {
+                  "field" : "_version",
+                  "value": [3, 6, 9]


Only a single version / version_type can be specified per document, so maybe it doesn't make sense to test version / version_type with append processor.

remove version/version type tests for append processor complete yml tests

PnPie · 2018-02-13T20:22:46Z

Hi @martijnvg,
Thanks for your response, I updated the PR and tested in yml file to set version_type to external (by default it should be internal), and verify if it's good or not.

And also in ./elasticsearch/rest-api-spec/src/main/resources/rest-api-spec/test/get/90_versions.yml, I see we test versions using GET request for this moment, and the versioning system is to control the update of document, isn't it ? Maybe we should test it rather by updating the document (index the same document) ? Because for a GET request, specifying the version_type doesn't really make sense, does it ?

martijnvg · 2018-02-14T17:20:34Z

Hey @PnPie, I think we just need to test whether we can set version and version_type in an ingest pipeline. So the following scenarios should be tested:

Create pipeline that just sets a _version to 1, index a document with this pipeline, which should fail.
Create pipeline that sets version to 1 and set version_type to external. Then index like in the previous scenario and all indexing requests should succeed.

That is it, no need test anything else.

martijnvg · 2018-02-16T06:36:30Z

@elasticmachine test this please

martijnvg · 2018-02-16T06:37:19Z

@PnPie If you like I can make these changes to the yaml test? Then we can get this change merged in.

PnPie · 2018-02-16T09:41:15Z

Hello @martijnvg,
Sorry for the late response, seems there are still some serialization problem for version_type according to the tests. And I also find that when we set version using set processor, I have an error of "cannot cast Integer to Long" because the value we set is considered as a Integer, but the version is a Long ?

And for the yml test, yes you can push directly on the branch, because maybe I didn't well get your point, the first scenario you mentioned, just set _version to 1, it should not fail ?

…_set

…out hassle. When applying the version to the IndexRequest just cast to Number and fetch its long value. Simplify yaml versions to just tests to see that both version and version_type can be used from an ingest pipeline.

martijnvg · 2018-02-16T15:58:42Z

@PnPie No problem. I addressed the issues that caused the build to fail in the latest commit. So hopefully we can get this PR in soon.

And for the yml test, yes you can push directly on the branch, because maybe I didn't well get your point, the first scenario you mentioned, just set _version to 1, it should not fail ?

Yes, so the first pipeline with use internal versioning. The first version of a document is always -2 I believe, so setting the expected version to 1 will make it fail. In case of external versioning ES is less strict when it comes to expecting what version it expected. This way we can see whether setting version type and version in a pipeline actually works.

martijnvg · 2018-02-16T15:58:51Z

@elasticmachine test this please

PnPie · 2018-02-16T16:57:03Z

Okay @martijnvg thx for the fix ! I just thought the initial version of a document is 1 ...

…_set

martijnvg · 2018-02-19T09:22:14Z

@elasticmachine test this please

martijnvg · 2018-02-20T09:18:34Z

@elasticmachine test this please

martijnvg · 2018-02-21T08:35:57Z

@PnPie Thank you for your contribution!

Add support version and version_type in ingest pipelines Add support for setting document version and version type in set processor of an ingest pipeline.

* es/master: (143 commits) Revert "Disable BWC tests for build issues" Remove AcknowledgedRestListener in favour of RestToXContentListener (#28724) Build: Consolidate archives and packages configuration (#28760) Skip some plugins service tests on Windows Migrate some *ResponseTests to AbstractStreamableXContentTestCase (#28749) Disable BWC tests for build issues Ensure that azure stream has socket privileges (#28751) [DOCS] Fixed broken link. Pass InputStream when creating XContent parser (#28754) [DOCS] Changed to use transient setting to reenabled allocation. Closes #27677 Delay path expansion on Windows [TEST] replace randomAsciiAlphanumOfLengthBetween with randomAsciiLettersOfLengthBetween [Tests] Extract the testing logic for Google Cloud Storage (#28576) [Docs] Update links to java9 docs (#28750) version set in ingest pipeline (#27573) Revert "Add startup logging for standalone tests" Tests: don't wait for completion while trying to get completed task Add 5.6.9 snapshot version [Docs] Java high-level REST client : clean up (#28703) Updated distribution outputs in contributing docs ...

* es/6.x: (140 commits) Remove AcknowledgedRestListener in favour of RestToXContentListener (#28724) Migrate some *ResponseTests to AbstractStreamableXContentTestCase (#28749) Revert "Disable BWC tests for build issues" Skip some plugins service tests on Windows Build: Consolidate archives and packages configuration (#28760) Ensure that azure stream has socket privileges (#28773) Disable BWC tests for build issues Pass InputStream when creating XContent parser (#28754) [DOCS] Fixed broken link. [DOCS] Changed to use transient setting to reenabled allocation. Closes #27677 Delay path expansion on Windows [TEST] replace randomAsciiAlphanumOfLengthBetween with randomAsciiLettersOfLengthBetween REST high-level client: add support for Rollover Index API (#28698) [Tests] Extract the testing logic for Google Cloud Storage (#28576) Moved Grok helper code to a separate Gradle module and let ingest-common module depend on it. version set in ingest pipeline (#27573) [Docs] Update links to java9 docs (#28750) Revert "Add startup logging for standalone tests" Tests: don't wait for completion while trying to get completed task Add 5.6.9 snapshot version ...

Add support version and version_type in ingest pipelines Add support for setting document version and version type in set processor of an ingest pipeline.

PnPie added 2 commits November 28, 2017 20:49

support document version set in ingest pipelines

4e1b507

Add support for setting document version and version type in set processor of an ingest pipeline.

remove wildcard imports

180d8c1

DaveCTurner added :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP v6.2.0 v7.0.0 labels Nov 28, 2017

change line width

590bd72

fix precommit

f84d796

fix more precommit

e42a8de

Merge branch 'master' of github.com:elastic/elasticsearch into ingest…

47c3733

…_pipeline_version_set

Merge branch 'master' of github.com:elastic/elasticsearch into ingest…

7981402

…_pipeline_version_set

colings86 added v6.3.0 and removed v6.2.0 labels Jan 22, 2018

rjernst requested a review from martijnvg January 26, 2018 23:44

martijnvg reviewed Jan 30, 2018

View reviewed changes

Merge branch 'master' of github.com:elastic/elasticsearch into ingest…

d93d6bb

…_pipeline_version_set

update

ee9a6df

PnPie commented Feb 10, 2018

View reviewed changes

martijnvg reviewed Feb 13, 2018

View reviewed changes

Update

4b1ad02

remove version/version type tests for append processor complete yml tests

martijnvg added 2 commits February 16, 2018 15:47

Merge remote-tracking branch 'es/master' into ingest_pipeline_version…

380fced

…_set

Merge remote-tracking branch 'es/master' into ingest_pipeline_version…

1e5c3f9

…_set

removed unneeded changes

8799107

martijnvg merged commit 7d8fb69 into elastic:master Feb 21, 2018

martijnvg added the >enhancement label Feb 21, 2018

martijnvg pushed a commit that referenced this pull request Feb 21, 2018

version set in ingest pipeline (#27573)

b868109

Add support version and version_type in ingest pipelines Add support for setting document version and version type in set processor of an ingest pipeline.

martijnvg mentioned this pull request Feb 21, 2018

Support for document versions in ingester pipelines #27242

Closed

jimczi added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

tahaderouiche mentioned this pull request Nov 30, 2021

[Ingest pipeline] Add metadata for version and modified date #81148

Open

version set in ingest pipeline #27573

version set in ingest pipeline #27573

Conversation

PnPie commented Nov 28, 2017 • edited Loading

elasticmachine commented Nov 28, 2017

DaveCTurner commented Nov 28, 2017

PnPie commented Nov 28, 2017

rjernst commented Dec 21, 2017

elasticmachine commented Dec 22, 2017

rjernst commented Dec 22, 2017

rjernst commented Dec 22, 2017

PnPie commented Dec 24, 2017

rjernst commented Jan 9, 2018

rjernst commented Jan 10, 2018

PnPie commented Jan 10, 2018 • edited Loading

rjernst commented Jan 16, 2018

PnPie commented Jan 16, 2018

rjernst commented Jan 24, 2018

martijnvg commented Jan 30, 2018

martijnvg left a comment

Choose a reason for hiding this comment

martijnvg Jan 30, 2018

Choose a reason for hiding this comment

martijnvg Jan 30, 2018

Choose a reason for hiding this comment

martijnvg Jan 30, 2018

Choose a reason for hiding this comment

PnPie commented Feb 10, 2018 • edited Loading

PnPie Feb 10, 2018

Choose a reason for hiding this comment

martijnvg Feb 13, 2018

Choose a reason for hiding this comment

martijnvg left a comment

Choose a reason for hiding this comment

martijnvg Feb 13, 2018

Choose a reason for hiding this comment

martijnvg Feb 13, 2018

Choose a reason for hiding this comment

PnPie commented Feb 13, 2018 • edited Loading

martijnvg commented Feb 14, 2018

martijnvg commented Feb 16, 2018

martijnvg commented Feb 16, 2018

PnPie commented Feb 16, 2018

martijnvg commented Feb 16, 2018

martijnvg commented Feb 16, 2018

PnPie commented Feb 16, 2018

martijnvg commented Feb 19, 2018

martijnvg commented Feb 20, 2018

martijnvg commented Feb 21, 2018

PnPie commented Nov 28, 2017 •

edited

Loading

PnPie commented Jan 10, 2018 •

edited

Loading

PnPie commented Feb 10, 2018 •

edited

Loading

PnPie commented Feb 13, 2018 •

edited

Loading