Using connection pools throws SocketException

[ryanvanderpol]

I'm upgrading an app from an older version of NEST to 1.0.0-rc1. One of the newer features I'm trying to use is connection pools. However, whenever I create a new client using either the StaticConnectionPool or the SniffingConnectionPool almost all actions against the client end up throwing a SocketException. Here is my initialization code:

var connectionPool = new StaticConnectionPool(new List<Uri>() {"<url.from.config>", "<url.from.config>"});
var settings = new ConnectionSettings(connectionPool, defaultIndex: "myindex");

var client = new ElasticClient(settings);

Later on I go and do the following:

client.CreateIndex("test");

And I get a SocketException. There appear to be a variety of error messages. Here are some of the common ones:

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

An operation was attempted on something that is not a socket

The system detected an invalid pointer address in attempting to use a pointer argument in a call

Here is a stack trace:

System.dll!System.Net.Sockets.Socket.Receive(byte[] buffer, int offset, int size, System.Net.Sockets.SocketFlags socketFlags) + 0x61 bytes  
System.dll!System.Net.Sockets.NetworkStream.Read(byte[] buffer, int offset, int size) + 0x83 bytes  
System.dll!System.Net.PooledStream.Read(byte[] buffer, int offset, int size) + 0x14 bytes   
System.dll!System.Net.Connection.SyncRead(System.Net.HttpWebRequest request, bool userRetrievedStream, bool probeRead) + 0xfc bytes 
System.dll!System.Net.ConnectStream.ProcessWriteCallDone(System.Net.ConnectionReturnResult returnResult) + 0x89 bytes   
System.dll!System.Net.HttpWebRequest.WriteCallDone(System.Net.ConnectStream stream, System.Net.ConnectionReturnResult returnResult) + 0xbb bytes    
System.dll!System.Net.ConnectStream.CallDone(System.Net.ConnectionReturnResult returnResult) + 0x63 bytes   
System.dll!System.Net.ConnectStream.CloseInternal(bool internalCall, bool aborting) + 0xde bytes    
System.dll!System.Net.ConnectStream.System.Net.ICloseEx.CloseEx(System.Net.CloseExState closeState) + 0x26 bytes    
System.dll!System.Net.ConnectStream.CloseInternal(bool internalCall) + 0x1a bytes   
System.dll!System.Net.HttpWebRequest.EndWriteHeaders_Part2() + 0x180 bytes  
System.dll!System.Net.HttpWebRequest.EndWriteHeaders(bool async) + 0x6c bytes   
System.dll!System.Net.HttpWebRequest.WriteHeadersCallback(System.Net.WebExceptionStatus errorStatus, System.Net.ConnectStream stream, bool async) + 0x14 bytes  
System.dll!System.Net.ConnectStream.WriteHeaders(bool async) + 0x2b5 bytes  
System.dll!System.Net.HttpWebRequest.EndSubmitRequest() + 0x8a bytes    
System.dll!System.Net.HttpWebRequest.SetRequestSubmitDone(System.Net.ConnectStream submitStream) + 0x11d bytes  
System.dll!System.Net.Connection.CompleteConnection(bool async, System.Net.HttpWebRequest request) + 0x16c bytes    
System.dll!System.Net.Connection.CompleteStartConnection(bool async, System.Net.HttpWebRequest httpWebRequest) + 0x167 bytes    
System.dll!System.Net.Connection.CompleteStartRequest(bool onSubmitThread, System.Net.HttpWebRequest request, System.Net.TriState needReConnect) + 0xed bytes   
System.dll!System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest request, bool forcedsubmit) + 0x3b3 bytes  
System.dll!System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest request, string connName) + 0x9b bytes   
System.dll!System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint servicePoint) + 0x173 bytes  
System.dll!System.Net.HttpWebRequest.GetResponse() + 0x33c bytes    
Elasticsearch.Net.dll!Elasticsearch.Net.Connection.HttpConnection.DoSynchronousRequest(System.Net.HttpWebRequest request, byte[] data, Elasticsearch.Net.Connection.Configuration.IRequestConfiguration requestSpecificConfig) Line 219 + 0xb bytes C#
Elasticsearch.Net.dll!Elasticsearch.Net.Connection.HttpConnection.HeaderOnlyRequest(System.Uri uri, string method, Elasticsearch.Net.Connection.Configuration.IRequestConfiguration requestSpecificConfig) Line 77 + 0x13 bytes C#
Elasticsearch.Net.dll!Elasticsearch.Net.Connection.HttpConnection.HeadSync(System.Uri uri, Elasticsearch.Net.Connection.Configuration.IRequestConfiguration requestSpecificConfig) Line 53 + 0x15 bytes C#
Elasticsearch.Net.dll!Elasticsearch.Net.Connection.Transport.Ping(Elasticsearch.Net.Connection.RequestState.ITransportRequestState requestState) Line 74 + 0x36 bytes   C#
Elasticsearch.Net.dll!Elasticsearch.Net.Connection.Transport.DoRequest<Nest.IndicesOperationResponse>(Elasticsearch.Net.Connection.RequestState.TransportRequestState<Nest.IndicesOperationResponse> requestState) Line 288 + 0xc bytes C#
Elasticsearch.Net.dll!Elasticsearch.Net.Connection.Transport.DoRequest<Nest.IndicesOperationResponse>(string method, string path, object data, Elasticsearch.Net.IRequestParameters requestParameters) Line 263 + 0x31 bytes    C#
Elasticsearch.Net.dll!Elasticsearch.Net.ElasticsearchClient.DoRequest<Nest.IndicesOperationResponse>(string method, string path, object data, Elasticsearch.Net.IRequestParameters requestParameters) Line 56 + 0x52 bytes  C#
Elasticsearch.Net.dll!Elasticsearch.Net.ElasticsearchClient.IndicesCreatePost<Nest.IndicesOperationResponse>(string index, object body, System.Func<Elasticsearch.Net.CreateIndexRequestParameters,Elasticsearch.Net.CreateIndexRequestParameters> requestParameters) Line 10581 + 0x3d bytes   C#
Nest.dll!Nest.RawDispatch.IndicesCreateDispatch<Nest.IndicesOperationResponse>(Nest.ElasticsearchPathInfo<Elasticsearch.Net.CreateIndexRequestParameters> pathInfo, object body) Line 1274 + 0xe5 bytes C#
Nest.dll!Nest.ElasticClient.CreateIndex.AnonymousMethod__f8(Nest.ElasticsearchPathInfo<Elasticsearch.Net.CreateIndexRequestParameters> p, Nest.ICreateIndexRequest d) Line 17 + 0x34 bytes  C#
Nest.dll!Nest.ElasticClient.Dispatch<Nest.ICreateIndexRequest,Elasticsearch.Net.CreateIndexRequestParameters,Nest.IndicesOperationResponse>(Nest.ICreateIndexRequest descriptor, System.Func<Nest.ElasticsearchPathInfo<Elasticsearch.Net.CreateIndexRequestParameters>,Nest.ICreateIndexRequest,Elasticsearch.Net.ElasticsearchResponse<Nest.IndicesOperationResponse>> dispatch) Line 84 + 0x11 bytes C#
Nest.dll!Nest.ElasticClient.CreateIndex(System.Func<Nest.CreateIndexDescriptor,Nest.CreateIndexDescriptor> createIndexSelector) Line 15 + 0x32 bytes    C#
Nest.dll!Nest.CreateIndexExtensions.CreateIndex(Nest.IElasticClient client, string index, System.Func<Nest.CreateIndexDescriptor,Nest.CreateIndexDescriptor> createIndexSelector) Line 22 + 0x2a bytes  C#
Evaluation of: client.CreateIndex("test")   

Doing the exact same thing, but NOT using a connection pool works just fine.

Here is the info from my ES instance:

{
    status: 200
    name: search2.localdomain
    version: {
        number: 1.1.1
        build_hash: f1585f096d3f3985e73456debdc1a0745f512bbc
        build_timestamp: 2014-04-16T14:27:12Z
        build_snapshot: false
        lucene_version: 4.7
    }
    tagline: You Know, for Search
}

[andersosthus]

Hi,

Just tried this code on my machine (ES 1.2.2, NEST 1.0.0-rc1), and I'm not getting any exceptions:

var connectionPool = new StaticConnectionPool(new List<Uri> { new Uri("http://localhost:9200") });
var client = new ElasticClient(new ConnectionSettings(connectionPool).SetDefaultIndex("nest_test_data-9700"));

Seems a bit strange though, are you able to reproduce it in a small console app, or is it just happening in that solution?

[ryanvanderpol]

Hey @andersosthus, thanks for the quick reply. I just set up a brand new blank console app and have tried to reproduce the issue. It is still not working, but now I am getting a MaxRetryException. Digging down through InnerExceptions doesn't seem to show much more than the same exception nested multiple times.

Failed after retrying 1 times: 'POST nest_test_pools_2'. InnerException: MaxRetryException, InnerMessage: Failed after retrying 1 times: 'POST nest_test_pools_2'. 
InnerException: MaxRetryException, InnerMessage: Failed after retrying 1 times: 'POST nest_test_pools_2'. , 
InnerStackTrace:    at Elasticsearch.Net.Connection.Transport.RetryRequest[T](TransportRequestState`1 requestState, Exception e) in c:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\Transport.cs:line 344
   at Elasticsearch.Net.Connection.Transport.DoRequest[T](TransportRequestState`1 requestState) in c:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\Transport.cs:line 291, InnerStackTrace:    
   at Elasticsearch.Net.Connection.Transport.RetryRequest[T](TransportRequestState`1 requestState, Exception e) in c:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\Transport.cs:line 344
   at Elasticsearch.Net.Connection.Transport.DoRequest[T](TransportRequestState`1 requestState) in c:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\Transport.cs:line 324
   at Elasticsearch.Net.Connection.Transport.RetryRequest[T](TransportRequestState`1 requestState, Exception e) in c:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\Transport.cs:line 346
   at Elasticsearch.Net.Connection.Transport.DoRequest[T](TransportRequestState`1 requestState) in c:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\Transport.cs:line 291

Again, switching the exact same code from using a StaticConnectionPool to hitting a single node works just fine.

[andersosthus]

Hi,

I remember I encountered the same exception earlier in the week while creating some tests in one of our solutions, connecting locally to a single node ES cluster running on a Small VM in Azure.

I ended up with this to create my client:

internal static ElasticClient CreateElasticClient(string index)
{
  var appSetting = ConfigurationHelper.GetMachineSpecificConfiguration("ElasticSearchUri"); 
  var node = new Uri(appSetting);
  var connectionPool = new StaticConnectionPool(new[] { node });
  var settings = new ConnectionSettings(connectionPool)
    .SniffOnConnectionFault(false)
    .SniffOnStartup(false)
    .SetTimeout(500)
    .SetPingTimeout(500)
    .MaximumRetries(5)
    .SetDefaultIndex(index);

  return new ElasticClient(settings);
}

Not sure if this will help you or not.

The ElasticClient we use in our test environment is something like this:

var node = new Uri(CloudConfigurationManager.GetSetting("ElasticSearchUri"));
var connectionPool = new SniffingConnectionPool(new[] {node});
var settings = new ConnectionSettings(connectionPool)
  .SniffOnConnectionFault()
  .SniffOnStartup()
  .SniffLifeSpan(TimeSpan.FromMinutes(1))
  .SetMaximumAsyncConnections(20)
  .MaximumRetries(5)
  .SetDefaultIndex("dummyindex");

return new ElasticClient(settings);

This works fine as well for our services, based on our initial tests.

Btw, are you running ES locally when you get this issue ?

[ryanvanderpol]

That initialization code seemed to have fixed the issue in my standalone console but not in my main app. I just find it strange that switching from a connection pool to a single URI causes it to work.

I am running against an ES cluster on EC2, not locally. My standalone console app is pointing at the exact same servers as my main app.

[andersosthus]

Hi,

Ok, you might have some other issues in your main app them, but a couple of things you could check:

• Make sure you're passing in proper Uri's. The first code you posted uses strings, but I assumed that was an oversight when sanitizing the urls before pasting here.
• If you're not using a SniffingConnectionPool, remember to set .SniffOnConnectionFault and SniffOnStartup to false, since they defaults to true
• I have not tried StaticConnectionPool with more than one node, but it should work fine and give you round robin. I'll test this at work tomorrow.
• In the RC there is a bug, so you must set DefaultIndex on a connection, or you could get some exceptions. This is fixed in the develop branch.

In your app are you doing anything special with the client as soon as it's created, as in DI or something that might complicate things ? We use Ninject to give us a Singleton ElasticClient with the settings above, and that works fine.

If you're still stuck, if I were you, I'd clone this repo and try to run the integration tests from the develop branch to see if they work. If so, you could reference the source directly in your app and step into the code to see if you can find what's going on.

Another suggestion would be to try the ElasticSearch.NET client directly:
http://nest.azurewebsites.net/elasticsearch-net/connecting.html and then try to do some request with that: http://nest.azurewebsites.net/elasticsearch-net/building-requests.html

Or you can wait until someone with more NEST experience comes along. They can probably provide better guidance than me :)

[ryanvanderpol]

Thanks a bunch for your help... Still not there, but getting closer.

I just switched it to use a SniffingConnectionPool with the code from your previous example and am still receiving the same error. Interestingly, this time it happened in the ElasticClient constructor, so it appears this error can occur before my client code even has a chance to do anything to it. Here's the error and a stack trace:

An operation was attempted on something that is not a socket 54.210.xx.xx:80

System.dll!System.Net.Sockets.Socket.DoConnect(System.Net.EndPoint endPointSnapshot, System.Net.SocketAddress socketAddress) + 0xd1 bytes   
System.dll!System.Net.ServicePoint.ConnectSocketInternal(bool connectFailure, System.Net.Sockets.Socket s4, System.Net.Sockets.Socket s6, ref System.Net.Sockets.Socket socket, ref System.Net.IPAddress address, System.Net.ServicePoint.ConnectSocketState state, System.IAsyncResult asyncResult, out System.Exception exception) + 0x1d1 bytes  
System.dll!System.Net.ServicePoint.GetConnection(System.Net.PooledStream PooledStream, object owner, bool async, out System.Net.IPAddress address, ref System.Net.Sockets.Socket abortSocket, ref System.Net.Sockets.Socket abortSocket6) + 0x102 bytes 
System.dll!System.Net.PooledStream.Activate(object owningObject, bool async, System.Net.GeneralAsyncDelegate asyncCallback) + 0x61 bytes    
System.dll!System.Net.Connection.CompleteStartConnection(bool async, System.Net.HttpWebRequest httpWebRequest) + 0xa6 bytes 
System.dll!System.Net.Connection.CompleteStartRequest(bool onSubmitThread, System.Net.HttpWebRequest request, System.Net.TriState needReConnect) + 0xed bytes   
System.dll!System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest request, bool forcedsubmit) + 0x3b3 bytes  
System.dll!System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest request, string connName) + 0x9b bytes   
System.dll!System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint servicePoint) + 0x173 bytes  
System.dll!System.Net.HttpWebRequest.GetResponse() + 0x33c bytes    
Elasticsearch.Net.dll!Elasticsearch.Net.Connection.HttpConnection.DoSynchronousRequest(System.Net.HttpWebRequest request, byte[] data, Elasticsearch.Net.Connection.Configuration.IRequestConfiguration requestSpecificConfig) Line 219 + 0xb bytes C#
Elasticsearch.Net.dll!Elasticsearch.Net.Connection.HttpConnection.HeadSync(System.Uri uri, Elasticsearch.Net.Connection.Configuration.IRequestConfiguration requestSpecificConfig) Line 53 + 0x21 bytes C#
Elasticsearch.Net.dll!Elasticsearch.Net.Connection.Transport.Ping(Elasticsearch.Net.Connection.RequestState.ITransportRequestState requestState) Line 74 + 0x1c bytes   C#
Elasticsearch.Net.dll!Elasticsearch.Net.Connection.Transport.DoRequest<System.IO.Stream>(Elasticsearch.Net.Connection.RequestState.TransportRequestState<System.IO.Stream> requestState) Line 289   C#
Elasticsearch.Net.dll!Elasticsearch.Net.Connection.Transport.Sniff(Elasticsearch.Net.Connection.RequestState.ITransportRequestState ownerState) Line 119 + 0x10 bytes   C#
Elasticsearch.Net.dll!Elasticsearch.Net.Connection.Transport.SniffClusterState(Elasticsearch.Net.Connection.RequestState.ITransportRequestState requestState) Line 145 + 0xa bytes  C#
Elasticsearch.Net.dll!Elasticsearch.Net.Connection.Transport.Transport(Elasticsearch.Net.Connection.IConnectionConfigurationValues configurationValues, Elasticsearch.Net.Connection.IConnection connection, Elasticsearch.Net.Serialization.IElasticsearchSerializer serializer, Elasticsearch.Net.Providers.IDateTimeProvider dateTimeProvider) Line 56 + 0xa bytes   C#
Elasticsearch.Net.dll!Elasticsearch.Net.ElasticsearchClient.ElasticsearchClient(Elasticsearch.Net.Connection.IConnectionConfigurationValues settings, Elasticsearch.Net.Connection.IConnection connection, Elasticsearch.Net.Connection.ITransport transport, Elasticsearch.Net.Serialization.IElasticsearchSerializer serializer) Line 40 + 0x22 bytes C#
Nest.dll!Nest.ElasticClient.ElasticClient(Nest.IConnectionSettingsValues settings, Elasticsearch.Net.Connection.IConnection connection, Nest.INestSerializer serializer, Elasticsearch.Net.Connection.ITransport transport) Line 50 + 0x28 bytes    C#

It may be worth noting that the nginx proxy on the server(s) has basic HTTP authentication enabled so my URLs are actually structured like this: http://username:[email protected] but since it works in my standalone console app, I assume this doesn't matter. Also, from the message text in the exception, the socket seems to be converting it to an IP address correctly anyway.

EDIT: I also converted the code to use the basic ElasticsearchClient instead of the NEST client and it throws the same error.

[andersosthus]

Shooting blind, but you could try to create your URI like this

var node = new UriBuilder();
node.Scheme = Uri.UriSchemeHttp;
node.UserName = "user";
node.Password = "pass";
node.Host = "localhost";
node.Port = 80;

[ryanvanderpol]

Nope. I also completely disabled the nginx basic auth on my servers and that didn't make any difference either, so I assume it's unrelated to that.

[ryanvanderpol]

I have fixed this, but I'm still not entirely sure what the cause was. In Visual Studio's NuGet Package Manager, I completely removed all references to NEST, cleaned the solution, closed VS, reopened and readded the NuGet package and it started working.

I'm guessing there was some assembly redirect or conflict perhaps? Hopefully something here can help anyone else who has the same problem.

[Mpdreamz]

Hi @ryanvanderpol many thanks for opening this PR

While its true that the RC should handle these exceptions properly, your PR did prompt me to add better unit tests for the following situations

• Sync call, ping throws exception at IConnection level.
• Sync call, sniff throws exception at IConnection level.
• Async call, ping throws exception at IConnection level.
• Async call, sniff throws exception at IConnection level.
• Async call, call itself throws exception at IConnection level.

All the async tests I wrote returned a task that will throw an exception as suppose to throwing an exception directly when called.

While the synchronous calls all behaved well (which you are using) the failover of these hard exceptions when calling async methods did not all properly caused a failover to happen.

this is now fixed #803

@andersosthus thanks for your sound help on this thread!

A couple of things I like to clarify:

• Nest and Elasticsearch.NET use the exact same ITransport so behave exactly the same.
• SniffOnConnectionFault and SniffOnStartup default to false. On top of that if an IConnectionPool signals it does not accept updates sniffing will never happen regardless of those 2 settings. StaticConnectionPool does not accept updates.
• .SetMaximumAsyncConnections(20) I personally only use this setting in my _bulk indexation routines. For most other use cases (searches etcetera) its probably best to leave it unbounded.

[andersosthus]

@Mpdreamz thanks for the clarifications. I guess I misunderstood some of those points from looking at the docs.

I'll see if I can't update the Connection part of the docs to specify the stuff about SniffOnConnectionFault and SniffOnStartup.

A small question regarding the .SetMaximumAsyncConnections(20) - I guess you set it to 20 during bulk so that you won't overload ES when bulk loading data, is this assumption correct?

[Mpdreamz]

Thats correct, the only time I manage to upset elasticsearch as suppose to the client is when too issueing too many bulk requests. The actual exception returned by elasticsearch in those cases being a threadpool starvation related exception.

Sometimes tweaking the threadpool settings for bulk requests helps to be able to up the concurrency but in general I would not advise changing this, it all depends on the machines/cluster and your cluster usage pattern. For example upping the bulk threadpool settings might push other threadpools to their maximum. A general rule of thumb is that the defaults elasticsearch settings comes with are golden, very seldom do they need to be touched :)

Thanks for the heads up that the docs are not clear about all of this. I would love a PR in your words as I'm approaching it not with fresh eyes.