Using a named pipeline with partial document updates

[brostad]

We've got an ingest pipeline in our cluster that we use for setting the processing time for each document as it gets indexed, this is very useful for tracing when that happened because the documents can arrive through a number of sub systems and not necessarily when they were written. Now I'm trying to use the same mechanism for setting a new processing time for a document if it gets a partial update.

However, from the perldoc and from my initial testing it seems impossible to pass a partial document to a pipeline for processing before indexing.

I've both tested the update() method with the 'pipeline' parameter, like I'm doing in index(), and the Bulk method add_action(), using pipeline=>$id, action => 'update', but both failed with "Unknown param (pipeline) in (update) request".

Is there a limitation in Elasticsearch that doesn't allow partial updates to be ingested in a pipeline or is it just a missing feature in Search::Elasticsearch?

If it's not possible to run a partial update through a pipeline, how is it possible to ensure that both add_action() for updates and partial documents sent with update() sets a new timestamp in the existing "processing_time" field of the updated document?

[clintongormley]

See elastic/elasticsearch#17895

[brostad]

Thank you for the quick reply! Looks like I'll have to look into scripting the updates in my middle layer :)