Allow invalid values to be ignored

[apatrida]

For datatypes that expect a specific format, allow a variation of the type that is a "softie" in that it allows bad things to just be ignored. This is important for unclean data, and although other work may allow you to write a document processing plugin to ES that can clean data (i.e. parse human readable dates into a standard form), it is sometimes likely you can have some rough data that you don't want to kill your indexing. Especially for fields that are not that important but you want a best attempt to insert the document.

You then could also consider marking documents that have validation errors so they could be later rescanned and reindexed given the stored JSON when a cleaner is added that would possibly resolve the problem. For example, mark the document as validation error for field XX and later search for those docs and ask the system "reindex document " from its own stored form.

So 3 things here:

• be able to mark a field mapping to allow invalid data to be ignored (discarded) on a field by field basis
• have the system mark the record as not passing validation at a per-field level
• be able to ask the system to reindex by ID using _stored JSON or even by reindex-by-query (Reindex from _source by document ID or Query #492)

[kimchy]

Currently, the only field type that qualifies this is the date type field. We can add something like that to it, and in case its fails to parse, just don't index it. You can always query for documents that don't have this field and then handle it? If the actual value still needs to be stored as well, it can be a multi_field mapping, one with plane string type, and one with date in this "soft" format.

[apatrida]

Numeric fields that don't parse wouldn't qualify (or are they handled in creating the JSON or previous?)

[kimchy]

Yea, json already handles it, since it has native types for numbers.

[apatrida]

dandy.

[clintongormley]

Also, there is now a flag for ignoring malformed values