5.1 Unable to use index templates to reindex into daily indexes.

[niemyjski]

From the docs (https://www.elastic.co/guide/en/elasticsearch/reference/5.1/docs-reindex.html#_reindex_daily_indices) it sounds like this is possible but I run into issues using an index template. This might also reference: #18654 and #20465

My issue is simple, I am upgrading our oss project from 1.x to 5.x using external reindex. We are moving from monthly indexes to daily indexes and I have no clue how many indexes should be created (this needs to run on everyones machine and we are calculating it with a inline script based on a doc's utc time). So an index template would work perfectly just for the reindex. The problem is, I get index not found exceptions on reindex...

PUT _template/events-v1
{
  "template": "events-v1-*"
  ........
}

POST _reindex?wait_for_completion=false
{
  "source": {
    "remote": {
      "host": "http://localhost:9201"
    },
    "index": "events-v1-*",
    "sort": { "date": "desc" }
  },
  "dest": {
    "index": "events-v1-error"
  },
  "script": {
    "lang": "painless",
    "inline": "ctx._index = 'events-v1-' + DateTimeFormatter.ofPattern('yyyy.MM.dd').format(OffsetDateTime.parse(ctx._source.date).toInstant().atZone(ZoneOffset.UTC));"
  }
}

results in millions and millions of these errors:

{
      "index": "events-v1-2016.12.30",
      "type": "events",
      "id": "586651aba5abbe018465d6b6",
      "cause": {
        "type": "index_not_found_exception",
        "reason": "no such index",
        "resource.type": "index_expression",
        "resource.id": "events-v1-2016.12.30",
        "index_uuid": "_na_",
        "index": "events-v1-2016.12.30"
      },
      "status": 404
    }

[niemyjski]

Doh, narrowed this down to action.auto_create_index: false I wonder if there is anything we can do to improve this error message?

[nik9000]

I'll have a look at improving the error message.

[niemyjski]

Also, seems like there should be a auto_create_index mode that creates indexes when they match a template, but otherwise it doesn't happen..

[jasontedor]

@niemyjski We have an open issue for that: #20640

[nik9000]

auto_create_index can be a pattern (see bottom of this). I think that might be enough, at least for now.

[nik9000]

auto_create_index can be a pattern (see bottom of this). I think that might be enough, at least for now.

Sorry, hadn't updated my window and didn't know about the issue.