Preventing watcher from starting up if its templates are missing (elastic#82395)

This commit prevents watcher from failing to start if its templates are unavailable. Previously the watcher service
would fail to start if (for example) the .watch-history-14 template did not exist. This could happen during a rolling
upgrade. In that case, the watcher service would fail to start even though it could reasonably keep writing without its
templates.
Relates elastic#69328 elastic#82109

Author: masseyke

x-pack/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/WatcherService.java

@@ -136,13 +136,6 @@ public class WatcherService {
      * @return true if everything is good to go, so that the service can be started
      */
     public boolean validate(ClusterState state) {
-        // template check makes only sense for non existing indices, we could refine this
-        boolean hasValidWatcherTemplates = WatcherIndexTemplateRegistry.validate(state);
-        if (hasValidWatcherTemplates == false) {
-            logger.debug("missing watcher index templates, not starting watcher service");
-            return false;
-        }
-
         IndexMetadata watcherIndexMetadata = WatchStoreUtils.getConcreteIndex(Watch.INDEX, state.metadata());
         IndexMetadata triggeredWatchesIndexMetadata = WatchStoreUtils.getConcreteIndex(
             TriggeredWatchStoreField.INDEX_NAME,
@@ -211,6 +204,10 @@ void stopExecutor() {
      * @param state cluster state, which is needed to find out about local shards
      */
     void reload(ClusterState state, String reason) {
+        boolean hasValidWatcherTemplates = WatcherIndexTemplateRegistry.validate(state);
+        if (hasValidWatcherTemplates == false) {
+            logger.warn("missing watcher index templates");
+        }
         // this method contains the only async code block, being called by the cluster state listener
         // the reason for this is, that loading he watches is done in a sync manner and thus cannot be done on the cluster state listener
         // thread