ES 2.2.0 delete by query plugin fails for data with external versioning

[natelapp]

Using 2.2.0, I am unable to delete by query for data that has been indexed using external_gte version type. Here's the error that I'm receiving:

[ec2-user@es1-dev ~]$ curl -XDELETE 'http://es1:9200/testindex/_query?q=repo:testing'
{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: illegal version value [0] for version type [INTERNAL];2: illegal version value [0] for version type [INTERNAL];3: illegal version value [0] for version type [INTERNAL];4: illegal version value [0] for version type [INTERNAL];5: illegal version value [0] for version type [INTERNAL];6: illegal version value [0] for version type [INTERNAL];7: illegal version value [0] for version type [INTERNAL];8: illegal version value [0] for version type [INTERNAL];9: illegal version value [0] for version type [INTERNAL];10: illegal version value [0] for version type [INTERNAL];"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: illegal version value [0] for version type [INTERNAL];2: illegal version value [0] for version type [INTERNAL];3: illegal version value [0] for version type [INTERNAL];4: illegal version value [0] for version type [INTERNAL];5: illegal version value [0] for version type [INTERNAL];6: illegal version value [0] for version type [INTERNAL];7: illegal version value [0] for version type [INTERNAL];8: illegal version value [0] for version type [INTERNAL];9: illegal version value [0] for version type [INTERNAL];10: illegal version value [0] for version type [INTERNAL];"},"status":400}

The delete by query succeeds for an index that doesn't use external_gte.

thanks!

[clintongormley]

@bleskes what do you think?

[bleskes]

This indeed an unfortunate case where internal and external versioning do not mix well. Internal version mean that ES is the source of truth for changes - it is incremented with every change in ES and starts with 1. External versioning assumes that some other system tracks document changes (including deletes). Originally 0 was an invalid value for external versioning but it wasn't enforced in code. When we fixed the latter people complained and we have changed semantics to allow 0 as a valid external value (see #5662). Now you can insert a value that's valid as an external version but is illegal for internal.

The delete by query plugin uses internal versioning to make sure the documents it deletes didn't change during it's operations. However, since the documents were indexed using the external versioning, their version is 0 which is illegal.

Can you tell us a bit more about your setup? Why are you using the delete by query plugin where you have some external source of truth? I would presume you would delete documents there first and have those propagated to ES as deletes with an external version?

[natelapp]

We receive our data from a third-party that supplies versions, starting with 0. For one of our indexes, we only care about the most recent version of a given resource, but need to be able to support reloading old data (mapping changes, etc). In order to ensure we're only keeping the latest (regardless of order received) we've gone with indexing using external_gte. Our process simply ignores the VersionConflictException that gets returned when attempting to add an older version. It has worked rather well for us.

Periodically, we'll need to delete data, for a variety of reasons. These are one-off deletes, usually related to expiring license agreements and such, and are separate from any versioning scheme. Historically we've just manually done a delete by query to handle these cases, which has served us well until recently.

[niemyjski]

I'm using internal indexing and hitting this on index...

illegal version value [0] for version type [INTERNAL];

[bleskes]

@niemyjski as we discussed in another issue, your issue is different than this one.

@natelapp thanks for the update. The problem is that currently doesn't align with the main use case for external versioning, where some external source owns all changes to the documents, including deletes. I haven't come up with a clean way of allowing you to do what you need plus making other use cases work without surprises. As a workaround for now, I think the easiest for you is to always +1 the version you get from your data source (to allow a delete by query operation).

[henningandersen]

@natelapp thanks for reporting this issue. The issue has been fixed in the upcoming 6.7 and 7.0 versions and I will therefore close this issue.