Watcher history created as index with wrong mapping instead of data stream after 8.0 upgrade

[henrikno]

Elasticsearch Version

8.0.1

Installed Plugins

No response

Java Version

bundled

OS Version

Elastic Cloud

Problem Description

After upgrading from 7.17 to 8.0.1 we noticed that some watches would not write their history.
The ES logs has many log lines similar to
[instance-0000000115] watch history could not be written [mywatch_5a0585ef-b8b1-4526-8b24-9fb9bf58981d-2022-03-15T11:29:32.801579308Z], failure [org.elasticsearch.index.mapper.MapperParsingException: failed to parse]

I noticed that .watcher-history-16 has the wrong mapping as everything just has default .text+.keyword, and the index template has a lot more specific fields. But the index template applies to a data stream, but .watcher-history-16 got created as an index, so the template didn't apply. It should have been created as a data stream after #64252, but somehow it didn't.

Steps to Reproduce

(unconfirmed)
Create a 7.17 cluster with multiple nodes
Create multiple watches that fire rapidly.
Upgrade ES to 8.x
.watcher-history-16 should be created as data stream, but got created as an index.

I tried it with a simple cluster and one watch firing every second and it upgraded to datastream, so it might be a race condition or depend on how many watches/how many nodes (we have ~80 watches/12 nodes)

Logs (if relevant)

No response

This looks a bit similar to #56732 except now that it's a data stream it won't be "fixed" by waiting until roll over.

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[jakelandis]

I believe this is caused by #82395 , which is a reaction to #82109. It's hard to follow the history, but there is indeed a race condition in the cluster for when Watcher can start and when all of the needed resources such as index templates are in place. We can't simply revert #82395 since it would re-introduce the worse side of this race condition - silently stopping Watcher until a manual restart or changes to the shard allocation.

Attempts to mitigate a related issue was introduced on #69147 and since upgrades now require 7.last I think the requirement to upgrade the template on the master node might be unnecessary and no longer requiring the master node might be the fix here. I assume (not verified) that this can only happen if a Watcher node is upgraded before the master node.

Not a great workaround ... but if you delete the index after the cluster is fully upgraded it will be recreated properly as the data stream with the correct mappings.