2.0.2

Bug Fixes

#1817 Do not serialize null or conditionless queries within collections
#1820 Fix NullReferenceException that occurred when serializing a conditionless BoolQuery
#1821 Max recursion depth was ignored when using AutoMap() from the put mapping API

2.0.1

Patch release to fix a sniffer parsing bug (#1811 as reported by @joelse)

GA Release 2.0.0

This marks the first GA release of our 2.0 client with well over a 1000 commits since 1.7.1 (the currently last released NEST version in the 1.x range).

Improvements

• #1788 Rename IElasticsearchClient to IElasticLowLevelClient to better specify intent
• #1795 Add SuffixExpressionVisitor for appending suffixes to lambda expressions
• #1801 .ConfigureAwait(false) all async/await methods
• #1794 add DebugInformation to responses and exceptions to aid in understanding problems with client calls

Bug Fixes

• #1789 Move highlight_query to HighlightField
• #1787 Fix generated XML comment documentation
• #1786 Move detect_noop back to the request body for an update request
• #1784 When specifying a format and extended_bounds on a date histogram aggregation, include the date_optional_time format to allow Elasticsearch to deserialize the bounds

Back to the drawing board

We took some time to go back to the drawing board. NEST is quite old (started in 2010) and not all of the choices that have accumulated in the code base make sense anymore.
So we stepped back to properly formalize how we see the lifetime of a call and worked off of that. Armed with the following diagram, we completely rewrote NEST's internals; The old Task Parallel Library (TPL) based code is now replaced with async/await, we have a much saner approach to exceptions and errors, and we expose enough information as an audit trail so you don't ever have to guess what went down during a call.

Our internals now also reflect this:

• IElasticClient exposes all the Elasticsearch API endpoints e.g client.Search this calls into ITransport's 2 methods Request and RequestAsync
• the default ITransport uses the passed in IRequestPipelineFactory to create an RequestPipeline which implements IPipeline.

This pipeline now handles all of the failover/sniffing/pinging logic and directly reflects the flow diagram.

We also simplified IConnection down just 2 methods. This means the outer edges are clean (ITransport and IConnection) and implementing your own should be really really simple. All of these (and also IMemoryStreamProvider and IDateTimeProvider) can be injected on the constructor of the client.

Test Framework

Another huge endeavour is the rework of our test framework. NEST 1.x was always well tested but used 5 different test projects and 5 years worth of changing our minds as how best to write tests and assertions, thus becoming a big hodgepodge of nunit assertions, fluent assertions, FakeItEasy, Moq combined with several different ways to compare json with object graphs and vice-versa. Trying to write a new test quickly became an exercise in yak shaving because there was no clear cut way on how best to write said test.

So the first thing we did as part of our 2.0 branch was to completely delete all of our tests. This act of insanity gave us carte blanche during our rewrite.

As of 2.0 we have one test project Tests and all of the tests are written in such a way that they can be run in unit test mode and integration test mode. Write once, run differently.
All of the API endpoint tests test all 4 request variations - two DSL's (fluent and object initializer) with both synchronous and asynchronous variations. We also test all of the moving parts of the Elasticsearch DSL (Aggregations, Sorting, IndexSettings, etc...) in the same way.

In addition to the more formal unit and integration tests, we also implemented a thing we dubbed Literate Testing to allow us to write tests in a more story telling form, with multi-line comments serving as the narrative for our asciidoc documentation while using the Roslyn compiler to pick out the interesting bits of code. This gives us the benefit of always compiling our documentation in addition to having only one place where we document, test and assert how a piece of code is supposed to work.

Another huge component of our testing framework is the Virtual Cluster that allows us to write tests for any situation and how we expect the client to behave.

/** we set up a 10 node cluster with a global request time out of 20 seconds.
* Each call on a node takes 10 seconds. So we can only try this call on 2 nodes
* before the max request time out kills the client call.
*/
var audit = new Auditor(() => Framework.Cluster
  .Nodes(10)
  .ClientCalls(r => r.FailAlways().Takes(TimeSpan.FromSeconds(10)))
  .ClientCalls(r => r.OnPort(9209).SucceedAlways())
  .StaticConnectionPool()
  .Settings(s => s.DisablePing().RequestTimeout(TimeSpan.FromSeconds(20)))
);

audit = await audit.TraceCalls(
  new ClientCall {
    { BadResponse, 9200 }, //10 seconds
    { BadResponse, 9201 }, //20 seconds
    { MaxTimeoutReached }
  },
  /**
  * On the second client call we specify a request timeout override to 80 seconds
  * We should now see more nodes being tried.
  */
  new ClientCall(r => r.RequestTimeout(TimeSpan.FromSeconds(80)))
  {
    { BadResponse, 9203 }, //10 seconds
    { BadResponse, 9204 }, //20 seconds
    { BadResponse, 9205 }, //30 seconds
    { BadResponse, 9206 }, //40 seconds
    { BadResponse, 9207 }, //50 seconds
    { BadResponse, 9208 }, //60 seconds
    { HealthyResponse, 9209 },
  }
);

This showcases the Virtual Cluster tests combined with Literate Tests and the extensive audit trail information available on each response (or exception).

I'm pleased to say we are back at a decent coverage rate (60%) and will continue to iterate and improve this.

Exception handling

Another big change in NEST 2.0 is how we deal with exceptions.

In NEST 1.x, the client threw a multitude of exceptions: MaxRetryException, ElasticsearchAuthException, ElasticsearchServerException, DslException, etc.. This made it challenging for users to handle exceptions and invalid responses, and understand the root cause of errors. On top of that, the types of exceptions thrown depended on what kind of IConnectionPool was injected, in order to maintain maximum backwards compatibility with NEST 0.x.

In NEST 2.x, exceptions are much more deterministic. The former ThrowOnElasticsearchServerExceptions() setting has been replaced with the more succinct ThrowExceptions(), which determines whether the client should ever throw an exception or not (client side and server exceptions). Furthermore, the types of exceptions have been reduced and simplified down to three types of exceptions:

ElasticsearchClientException: These are known exceptions, either an exception that occurred in the request pipeline(such as max retries or timeout reached, bad authentication, etc...) or Elasticsearch itself returned an error (could not parse the request, bad query, missing field, etc...). If it is an Elasticsearch error, the ServerError property on the response will contain the the actual error that was returned. The inner exception will always contain the root causing exception.

UnexpectedElasticsearchClientException: These are unknown exceptions, for instance a response from Elasticsearch not properly deserialized. These are usually bugs in the client and we encourage you to report them. This exception also inherits from ElasticsearchClientException so an additional catch block isn't necessary to handle but can be helpful in distinguishing between the two.

Runtime exceptions: These are CLR exceptions like ArgumentException, NullArgumentException etc. that are thrown when an API in the client is misused.

Breaking Changes

Even though a lot of work went into the interior, the exterior did not escape unscathed! On top of the many breaking changes that Elasticsearch 2.0 introduces, there are more then a few NEST 2.0 introduces.
We revalidated all the request and response domain objects against Elasticsearch 2.0.

A pretty complete list of breaking changes are available:

• Low level client changes, Elasticsearch.NET
• high level client changes, NEST

Elasticsearch 2.x support

NEST 2.0 supports all the new features in Elasticsearch 2.0 including pipeline aggregations. New features from Elasticsearch 2.2 have not yet been mapped.

Here we'll just highlight a couple features that are reflected in the NEST changes to whet your appetite!

Filters are Gone Gone Gone

In Elasticsearch 2.0, query and filter constructs have merged into one concept called queries and NEST 2.0 reflects this. So if you were previously using the AndFilter, this is now called the AndQuery; beware that some of these filters have been obsoleted and chances are high you were using them wrong!.

Isolated descriptors

In NEST 1.x we took pride in being a 1 to 1 mapping with the Elasticsearch API. In some cases however, this purity hid the real depth of parameters. As an example,

in NEST 1.x you could add sorts on Search() using:

client.Search<Project>(s=>s
  .SortAscending(...)
  .SortScript(...)
)

in NEST 2.0 you have to stoop down a level first in order to access the same functionality

client.Search<Project>(s=>s
  .Sor...

2.0.0-rc1

Release Candidate

This is the release candidate for NEST 2.x! Many improvements and fixes went into this release, including providing support for two new target frameworks, dotnet5.1 (.NETPlatform5.1) and net46 (.NETFramework4.6). As always, your feedback on this release are very welcome!

Improvements

• #1697 SUPPORT .NET CORE WITH DNX TOOLING
• #1089 Refactor to use System.Linq.Expressions.ExpressionVisitor
• #1686 Support other_bucket and other_bucket_key in filters aggregations
• #1705 AutoMap/Infer TimeSpan properties as long ticks
• #1742 Support looping of named aggregation buckets
• #1755 Add .NET 4.6 as a target framework

Bug Fixes

• #985 Reindex supports _parent and _routing
• #1433 Reindex supports reindexing one/many/all types within the from index
• #1490 Update now supports fields and is not generic
• #1612 Unify the methods and overloads available on Query<T>
• #1634 Ensure consistent purified Uris across all supported platforms
• #1708 Property Name should resolve to the last token
• #1729 Add Aggregation Metadata
• #1731 Use custom JsonConverters when supplied on derive JsonNetSerializer
• #1763 Add _timestamp to IHit<T>
• #1763 Add _ttl to IHit<T>
• #1766 when passed an a null reference for string, return null reference for DateMath
• #1767 Fields should not be null on response when specified on request
• #1772 Fix deficiencies in the dnx tooling for supporting dependency version ranges
• #1778 Align request/response type names
• #1776 Rename types in line with Naming Conventions
• #1773 Make ConnectionSettings protected on JsonNetSerializer

2.0.0-alpha2

Bug Fixes

• #1694 Add support for Newtosoft.Json 8
• #1706 Separate types for histogram and date histogram items
• #1709 Boolean and "conditionless" queries
• #1712 Rename ConnectionStatusHandler to OnRequestCompleted and add back functionality
• #1713 High level methods now take Time instead of TimeSpan
• #1720 SniffingConnectionPool now implements IDisposable (ty @garymcleanhall)
• #1726 Improved Norms implementation
• #1728 Add support for range queries on string fields
• #1736 Fix choosing the correct HTTP method when indexing an object with no Id
• #1741 Fix snowball analyzer language case sensitivity

2.0.0-alpha1

This marks the first release of our 2.0 branch (now master) with well over a 1000 commits since 1.7.1 (the currently last released NEST version in the 1.x range).

We are releasing it as alpha because we currently only support .NET 4.5 and up. For us to go from alpha to beta we ideally still have to finish

• PCL/.NET CORE are in the works but not part of our build and CI infrastructure.
• Finish our new documentation

Please note that even though this is an alpha, we have fully tested this against Elasticsearch 2.0 up to 2.1 (the current latest version).

Back to the drawing board

We took sometime to go back to the drawing board. NEST is quite old (started in 2010) and not all of the choices that have accumulated in the code base make sense anymore.
So we stepped back, formulized properly how we see the lifetime of a call, and worked off of that. Armed with the below diagram, we completely rewrote NEST's internals. The old TPL based code base is now replaced with async/await, we have a much saner approach to exceptions and errors, and we expose enough information as an audit trail so you do not have to ever guess what went down during a call.

Our internals now also reflect this:

IElasticClient exposes all the Elasticsearch API endpoints e.g client.Search this calls into ITransport's 2 methods Request and RequestAsync the default ITransport then uses the passed in IRequestPipelineFactory to create an RequestPipeline which implements IPipeline.

This pipeline now handles all of the failover/sniffing/pinging logic and directly reflects the flow diagram.

We also simplified IConnection down just 2 methods. This means the outer edges are clean (ITransport and IConnection) and implementing your own should be really really simple. All of these (and also IMemoryStreamProvider and IDateTimeProvider) can be injected on the constructor of the client.

Test Framework

Another huge endeavour is the rework of our test framework. NEST 1.x was always well tested but used 5 different test projects and 5 years worth of changing our minds as to how best to write tests and assertions.
Thus becoming a big hodgepodge of nunit assertions, fluent assertions, FakeItEasy, Moq combined with several different ways to compare json with object graphs and vice-versa. Trying to write a new test quickly became cumbersome because there was no clear cut way how best to write said test.

So the first thing we did as part of our 2.0 branch was to completely delete all of our tests. This act of insanity gave us carte blanche during our rewrite.

As of 2.0 we have one test project Tests and all of the tests are written in such a way that they can be run in unit test mode and integration test mode. Write once, run differently.
All the API endpoint tests test all 4 variations, 2 DSL's (fluent and object initializer) + sync and async. We also test all of the moving parts of Elasticsearch DSL (Aggregations, Sorting, IndexSettings, etc...) in the same way.

We also introduced a thing we dubbed Literate Testing to allow us to write tests in a more story telling form with the comments serving as the asciidoc for our documentation while using the Roslyn compiler to pick the interesting bits of code.
This gives us the benefit of always compiling our documentation but also having one place where we document, test and assert how a piece of code is supposed to work.

Another huge component of our testing framework is the Virtual Cluster that allows us to write tests for any situation and how we expect the client to behave.

/** we set up a 10 node cluster with a global request time out of 20 seconds.
* Each call on a node takes 10 seconds. So we can only try this call on 2 nodes
* before the max request time out kills the client call.
*/
var audit = new Auditor(() => Framework.Cluster
  .Nodes(10)
  .ClientCalls(r => r.FailAlways().Takes(TimeSpan.FromSeconds(10)))
  .ClientCalls(r => r.OnPort(9209).SucceedAlways())
  .StaticConnectionPool()
  .Settings(s => s.DisablePing().RequestTimeout(TimeSpan.FromSeconds(20)))
);

audit = await audit.TraceCalls(
  new ClientCall {
    { BadResponse, 9200 }, //10 seconds
    { BadResponse, 9201 }, //20 seconds
    { MaxTimeoutReached }
  },
  /**
  * On the second client call we specify a request timeout override to 80 seconds
  * We should now see more nodes being tried.
  */
  new ClientCall(r => r.RequestTimeout(TimeSpan.FromSeconds(80)))
  {
    { BadResponse, 9203 }, //10 seconds
    { BadResponse, 9204 }, //20 seconds
    { BadResponse, 9205 }, //30 seconds
    { BadResponse, 9206 }, //40 seconds
    { BadResponse, 9207 }, //50 seconds
    { BadResponse, 9208 }, //60 seconds
    { HealthyResponse, 9209 },
  }
);

This showcases the Virtual Cluster tests combined with Literate Tests and the extensive audit trail information available on each response (or exception).

I'm pleased to say we are back at a decent coverage rate (60%) and we'll continue to bump that.

Exception handling

Another big change in NEST 2.0 is how we deal with exceptions.

In NEST 1.x, the client threw a multitude of exceptions: MaxRetryException, ElasticsearchAuthException, ElasticsearchServerException, DslException, etc.. This made it challenging for users to handle exceptions/invalid responses, and understand the root cause of errors. On top of that, it depended on what kind of IConnectionPool was injected to contain maximum backwards compaitibility with NEST 0.x

In NEST 2.x, exceptions are much more deterministic. The former ThrowOnElasticsearchServerExceptions() setting has been replaced with simply ThrowExceptions(), which determinse whether the client should ever throw an exception or not (client side and server exceptions). Furthermore, the types of exceptions have been reduced and simplified-the client will now only throw three types of exceptions:

ElasticsearchClientException: These are known exceptions, either an exception that occurred in the request pipeline(such as max retries or timeout reached, bad authentication, etc...) or Elasticsearch itself returned an error (could not parse the request, bad query, missing field, etc...). If it is an Elasticsearch error, the ServerError property on the response will contain the the actual error that was returned. The inner exception will always contain the root causing exception.

UnexpectedElasticsearchClientException: These are unknown exceptions, for instance a response from Elasticsearch not properly deserialized. These are usually bugs and should be reported. This exception also inherits from ElasticsearchClientException so an additional catch block isn't necessary, but can be helpful in distinguishing between the two.

Development time exceptions: These are CLR exceptions like ArgumentException, NullArgumentException etc., that are thrown when an API in the client is misused. These should not be handled as you want to know about them during development.

Breaking Changes

Even though a lot of work went into the interior, the exterior did not escape unscathed! On top of the many breaking changes that Elasticsearch 2.0 introduces, there are more then a few NEST 2.0 introduces.
We revalidated all the request and response domain objects against Elasticsearch 2.0.

We will do our best to compile a list when NEST 2.0 GA hits. If we moved your cheese to a spot you can no longer find it then please open an issue and we'll be more then happy to help locate it.

Elasticsearch 2.x support

NEST 2.0 supports all the new features from Elasticsearch 2.0 including pipeline aggregations. Here we'll just highlight a couple features that are reflected in NEST changes

Removal of filters

NEST 2.0 reflects Elasticsearch 2.0 move of filters and no longer has filter constructs in its code base

Filtered query deprecation

With the removal of filters NEST has added a special construct in its Query DSL to easily create a bool query with a filter clause

.Query(q=> +q.Term(p=>p.Name, "NEST"))

Note the + this will wrap the term query inside a bool query's filter clause

You can even combine this with !

.Query(q=> !+q.Term(p=>p.Name, "NEST"))

This will wrap the term query inside a bool filter and that bool inside a bool must_not, obviously this also works for the object initializer syntax

!+new TermQuery {}

Attribute based mapping

The single ElasticPropertyAttribute has been broken up into individual attributes per property type.

For instance, the following:

[ElasticType(Name = "othername", IdProperty = "MyId")]
public class Foo
{
  [ElasticProperty(Type = FieldType.String)]
  public Guid MyId { get; set; }

  [ElasticProperty(Type = FieldType.String)]
  public string Name { get; set; }

  [ElasticProperty(Type = FieldType.String, Analyzer = "myanalyzer", TermVector = TermVectorOption.WithOffsets)]
  public string Description { get; set; }

  [ElasticProperty(Type = FieldType.Date, Format = "mmmddyyyy")]
  public DateTime Date { get; set; }

  [ElasticProperty(Type = FieldType.Integer, Coerce = true)]
  public int Number { get; set; }

  [ElasticProperty(Type = FieldType.Nested, IncludeInParent = true)]
  public List<Bar> Bars { get; set; }
}

becomes

[ElasticsearchType(Name = "othername", IdProperty = "MyId")]
public class Foo
{
  [String]
  public Guid MyId { get; set; }

  [String]
  public string Name { get; set; }

  [String(Analyzer = "myanalyzer", TermVector = TermVectorOption.WithOffsets)]
  public string Description { get; set; }

  [Date(Format = "mmddyyyy")]
  public DateTime Date { get; set; }

  [Number(NumberType.Integer, Coerce = true, DocValues = true)]
  public int Number { get; set; }

  [Nested(IncludeInParent = true)]
  public...

1.7.1

Bug Fixes

• #1533 Added missing store option to _timestamp field (ty @KodrAus !)
• #1525 Added missing offset option to date_histogram aggregation
• #1514 Added missing filter option to has_parent filter
• #1505 Added missing min and max options to has_child filter
• #1497 Fixed serialization of fields on UpdateDescriptor when using expressions

1.7.0

This release contains full feature parity with Elasticsearch 1.7.

New Features

• #1381 MapFromAttributes() now defaults Guids to strings
• #1454 Add Query(IQueryContainer) to CountDescriptor
• #1479 byte and short types are now mapped to their respective Elasticsearch types when using MapFromAttributes()
• #1485 Add support for version to snapshot info
• #1487 Add shard_size to TermStats face (TY @qbast)

Bug Fixes

• #1296 UpdateReponse was missing Fields
• #1440 Better syntax for ReIndex
• #1474 GeoBounds aggregation not read to completion
• #1478 Fixed serialization of LanguageAnalyzers stem_exclusion property
• #1484 Document metadata missing on TermVectorResponse
• #1496 Force invariant culture on all dates

1.6.1

• NEST upgraded to Json.NET 7.x
• #1463 Fix ScriptFile and ScriptId not being serialized on bulk update requests
• e344795 Fix boost option missing from several queries

1.6.0

This release contains full feature parity with Elasticsearch 1.6.

New Features

• #1446 Support for field stats API
• #1445 Support for synced flush API
• #1444 Support for response filtering on all APIs
• #1441 Support for murmur3 Field Type
• #1430 Support for include/exclude on TermsAggregation
• #1451 Added node id to _cat API responses
• #1450 Added default option to FieldValueFactor
• #1449 Added numeric_resolution to date mapping
• #1447 Added explanation to validate API response
• $1448 Added SortMulti() to allow for easy sorting across multiple fields (TY @mgoodfellow !)

Bug Fixes

• #1459, #1456 Add support for script, script id, and script file where they were missing on all update descriptors
• #1457 Fixed NullReferenceException in GetIndex() when a custom analyzer had no type defined
• #1438 GeoPoint mapping was missing precision
• #1429 DateMapping, GeoPointMapping, and BooleanMapping were missing doc_values
• #1409 Raw filters were not written to request when used on filtered aliases
• #1452 Fixed serialization of KeepWordsPath property on KeepWordsTokenFilter (TY @davidtme !)
• #1431 MoreLikeThis query should accept documents or ids (TY @robertlyson !)