Search.delete needs to handle kwargs

[ericandrewmeadows]

Search.delete does not take in keywords, such as conflicts=proceed, which is essential when trying to delete multiple objects, but currently the issue returned is:

ConflictError(409, u'{"took":1,"timed_out":false,"total":2,"deleted":0,"batches":1,"version_conflicts":2,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":[{"index":"mlsdata_test","type":"doc","id":"DlyKuWgBOtcMH22wMIjN","cause":{"type":"version_conflict_engine_exception","reason":"[doc][DlyKuWgBOtcMH22wMIjN]: version conflict, current version [2] is different than the one provided [1]","index_uuid":"YY-DGvIkSe6bGSQsobnYCA","shard":"0","index":"mlsdata_test"},"status":409},{"index":"mlsdata_test","type":"doc","id":"D1yKuWgBOtcMH22wMIjv","cause":{"type":"version_conflict_engine_exception","reason":"[doc][D1yKuWgBOtcMH22wMIjv]: version conflict, current version [2] is different than the one provided [1]","index_uuid":"YY-DGvIkSe6bGSQsobnYCA","shard":"0","index":"mlsdata_test"},"status":409}]}')
Traceback (most recent call last):
File "", line 186, in upsert_json_mls_listing_to_document
File "", line 147, in handle_previously_matched_record
File "/mnt/yarn/usercache/livy/appcache/application_1548450731252_0176/spark-eb780ca6-de11-4c8f-a6d1-449f4b5c3168/userFiles-2fa51379-ad00-48df-a497-ede1424836b5/elasticsearch_dsl.zip/elasticsearch_dsl/search.py", line 742, in delete
**self._params
File "/mnt/yarn/usercache/livy/appcache/application_1548450731252_0176/spark-eb780ca6-de11-4c8f-a6d1-449f4b5c3168/userFiles-2fa51379-ad00-48df-a497-ede1424836b5/elasticsearch.zip/elasticsearch/client/utils.py", line 76, in _wrapped
return func(*args, params=params, **kwargs)
File "/mnt/yarn/usercache/livy/appcache/application_1548450731252_0176/spark-eb780ca6-de11-4c8f-a6d1-449f4b5c3168/userFiles-2fa51379-ad00-48df-a497-ede1424836b5/elasticsearch.zip/elasticsearch/client/init.py", line 886, in delete_by_query
doc_type, '_delete_by_query'), params=params, body=body)
File "/mnt/yarn/usercache/livy/appcache/application_1548450731252_0176/spark-eb780ca6-de11-4c8f-a6d1-449f4b5c3168/userFiles-2fa51379-ad00-48df-a497-ede1424836b5/elasticsearch.zip/elasticsearch/transport.py", line 318, in perform_request
status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
File "/mnt/yarn/usercache/livy/appcache/application_1548450731252_0176/spark-eb780ca6-de11-4c8f-a6d1-449f4b5c3168/userFiles-2fa51379-ad00-48df-a497-ede1424836b5/elasticsearch.zip/elasticsearch/connection/http_requests.py", line 90, in perform_request
self._raise_error(response.status_code, raw_data)
File "/mnt/yarn/usercache/livy/appcache/application_1548450731252_0176/spark-eb780ca6-de11-4c8f-a6d1-449f4b5c3168/userFiles-2fa51379-ad00-48df-a497-ede1424836b5/elasticsearch.zip/elasticsearch/connection/base.py", line 125, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
ConflictError: ConflictError(409, u'{"took":1,"timed_out":false,"total":2,"deleted":0,"batches":1,"version_conflicts":2,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":[{"index":"mlsdata_test","type":"doc","id":"DlyKuWgBOtcMH22wMIjN","cause":{"type":"version_conflict_engine_exception","reason":"[doc][DlyKuWgBOtcMH22wMIjN]: version conflict, current version [2] is different than the one provided [1]","index_uuid":"YY-DGvIkSe6bGSQsobnYCA","shard":"0","index":"mlsdata_test"},"status":409},{"index":"mlsdata_test","type":"doc","id":"D1yKuWgBOtcMH22wMIjv","cause":{"type":"version_conflict_engine_exception","reason":"[doc][D1yKuWgBOtcMH22wMIjv]: version conflict, current version [2] is different than the one provided [1]","index_uuid":"YY-DGvIkSe6bGSQsobnYCA","shard":"0","index":"mlsdata_test"},"status":409}]}')

[honzakral]

You can always call .params(conflits="proceed") before calling .delete() or .extra(conflicts="proceed") to include it in the body.

Hope this helps!

[shivam05011996]

@honzakral I tried the above the solution but it does not seem to help.
here is the function I with whose help I am trying to perform delete_by_query -

def unindex(_id, index):
    query = unindex_query(_id)

    s = Search(index=index).query(Q(query))
    # For travis tests to pass
    s.params(conflicts='proceed')
    s.delete()

query looks something like this -

query = {'match': {'_id': 'test123'}}

(Pdb) Q(query)
Match(_id='test123')

Is there something wrong the way I have Implemented?

ES version and elasticsearch_dsl_version 6.6

[honzakral]

you need to assign back to s: s = s.params(...) since params also returns a clone of the Search object.

[shivam05011996]

@honzakral The above solution is something like, skipping the deletion operation if I am correct because the record does not gets deleted rather it creates a duplicate one. Have a look at screenshot -

Ideally, the total record should have been empty because there will be a tearDown after every test. Also the _id values should not have been more than 3 if its deleting everything in tearDown.

[vermavikrant]

@honzakral @shivam05011996 I am facing similar problem. Before I go ahead and rely on s = s.params(conflicts='proceed') I wanted to confirm that even if in response we see version_conflicts greater than 0, internally & eventually elastic will delete these conflict documents as well. Is that correct?

• if yes, then the msg post from @shivam05011996 confused me that it wont delete the conflict docs when conflicts='proceed'.
• if no, then it means i should keep retrying with conflicts='proceed' until we see version_conflicts as 0 in response. ?