Skip to content

Starting multiple nodes at the same time can cause the cluster not to communicate properly #102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
clintongormley opened this issue Apr 1, 2010 · 5 comments
Closed

Starting multiple nodes at the same time can cause the cluster not to communicate properly #102

clintongormley opened this issue Apr 1, 2010 · 5 comments

Comments

@clintongormley
Copy link
Contributor

Hiya

I've had a repeat occurrence of two issues (1) starting all 3 nodes at the same time means that the cluster doesn't communicate properly and (2) starting one node doesn't read the index in the gateway, but starting a second node does.

When I start a node, it puts a lot of load (loads of 30+) on the CPU. Sometimes, if I start (eg) 3 nodes at the same time, they fail to communicate properly, possibly because the high CPU loads are causing timeouts?

That's what happened here - 2 nodes were talking, and the third wasn't. I noticed at this stage that there were two metadata files in my gateway directory.

I shut down all 3 nodes, then restarted the first node. It didn't read the existing index from the gateway. There was nothing in the log about having found an index, and the work directory didn't grow in size (as it usually does while it builds the niofs caches).

I started the second node, and then it did find the index, and everything works correctly.

Below is the metadata file just before the second restart, plus the logs from all 3 servers:

METADATA:
{
"meta-data" : {
"maxNumberOfShardsPerNode" : 100,
"indices" : {
"ia_object_1270054545" : {
"settings" : {
"index.analysis.analyzer.default.filter.1" : "lowercase",
"index.analysis.analyzer.default.filter.2" : "stop",
"index.numberOfReplicas" : "2",
"index.analysis.analyzer.default.filter.0" : "standard",
"index.analysis.analyzer.default.filter.3" : "asciifolding",
"index.analysis.analyzer.default.tokenizer" : "standard",
"index.numberOfShards" : "5",
"index.aliases.0" : "ia_object"
},
"mappings" : {
"site" : {
"source" : "{\n "site" : {\n "type" : "object",\n "dynamic" : true,\n "enabled" : true,\n "pathType" : "full",\n "dateFormats" : [ "dateOptionalTime" ],\n "boostField" : {\n "name" : "_boost"\n },\n "allField" : {\n "enabled" : true,\n "store" : "yes",\n "termVector" : "with_positions_offsets"\n },\n "sourceField" : {\n "name" : "_source",\n "enabled" : true\n },\n "properties" : {\n "id" : {\n "type" : "long",\n "indexName" : "id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "ancestor_ids" : {\n "type" : "long",\n "indexName" : "ancestor_ids",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "last_modified" : {\n "type" : "date",\n "indexName" : "last_modified",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "status" : {\n "type" : "string",\n "indexName" : "status",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "created" : {\n "type" : "date",\n "indexName" : "created",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "creator_id" : {\n "type" : "long",\n "indexName" : "creator_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "nullValue" : 0\n },\n "parent_id" : {\n "type" : "long",\n "indexName" : "parent_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n }\n }\n }\n}"
},
"source" : {
"source" : "{\n "source" : {\n "type" : "object",\n "dynamic" : true,\n "enabled" : true,\n "pathType" : "full",\n "dateFormats" : [ "dateOptionalTime" ],\n "boostField" : {\n "name" : "_boost"\n },\n "allField" : {\n "enabled" : true,\n "store" : "yes",\n "termVector" : "with_positions_offsets"\n },\n "sourceField" : {\n "name" : "_source",\n "enabled" : true\n },\n "properties" : {\n "id" : {\n "type" : "long",\n "indexName" : "id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "ancestor_ids" : {\n "type" : "long",\n "indexName" : "ancestor_ids",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "last_modified" : {\n "type" : "date",\n "indexName" : "last_modified",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "created" : {\n "type" : "date",\n "indexName" : "created",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "status" : {\n "type" : "string",\n "indexName" : "status",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "name" : {\n "type" : "string",\n "indexName" : "name",\n "index" : "analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "creator_id" : {\n "type" : "long",\n "indexName" : "creator_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "nullValue" : 0\n },\n "url" : {\n "type" : "string",\n "indexName" : "url",\n "index" : "analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "parent_id" : {\n "type" : "long",\n "indexName" : "parent_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n }\n }\n }\n}"
},
"scraper" : {
"source" : "{\n "scraper" : {\n "type" : "object",\n "dynamic" : true,\n "enabled" : true,\n "pathType" : "full",\n "dateFormats" : [ "dateOptionalTime" ],\n "boostField" : {\n "name" : "_boost"\n },\n "allField" : {\n "enabled" : true,\n "store" : "yes",\n "termVector" : "with_positions_offsets"\n },\n "sourceField" : {\n "name" : "_source",\n "enabled" : true\n },\n "properties" : {\n "id" : {\n "type" : "long",\n "indexName" : "id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "ancestor_ids" : {\n "type" : "long",\n "indexName" : "ancestor_ids",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "last_modified" : {\n "type" : "date",\n "indexName" : "last_modified",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "created" : {\n "type" : "date",\n "indexName" : "created",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "status" : {\n "type" : "string",\n "indexName" : "status",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "name" : {\n "type" : "string",\n "indexName" : "name",\n "index" : "analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "creator_id" : {\n "type" : "long",\n "indexName" : "creator_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "nullValue" : 0\n },\n "parent_id" : {\n "type" : "long",\n "indexName" : "parent_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n }\n }\n }\n}"
},
"location" : {
"source" : "{\n "location" : {\n "type" : "object",\n "dynamic" : true,\n "enabled" : true,\n "pathType" : "full",\n "dateFormats" : [ "dateOptionalTime" ],\n "boostField" : {\n "name" : "_boost"\n },\n "allField" : {\n "enabled" : true,\n "store" : "yes",\n "termVector" : "with_positions_offsets"\n },\n "sourceField" : {\n "name" : "_source",\n "enabled" : true\n },\n "properties" : {\n "id" : {\n "type" : "long",\n "indexName" : "id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "ancestor_ids" : {\n "type" : "long",\n "indexName" : "ancestor_ids",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "last_modified" : {\n "type" : "date",\n "indexName" : "last_modified",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "created" : {\n "type" : "date",\n "indexName" : "created",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "status" : {\n "type" : "string",\n "indexName" : "status",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "name" : {\n "type" : "string",\n "indexName" : "name",\n "index" : "analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "creator_id" : {\n "type" : "long",\n "indexName" : "creator_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "nullValue" : 0\n },\n "parent_id" : {\n "type" : "long",\n "indexName" : "parent_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n }\n }\n }\n}"
},
"image" : {
"source" : "{\n "image" : {\n "type" : "object",\n "dynamic" : true,\n "enabled" : true,\n "pathType" : "full",\n "dateFormats" : [ "dateOptionalTime" ],\n "boostField" : {\n "name" : "_boost"\n },\n "allField" : {\n "enabled" : true,\n "store" : "yes",\n "termVector" : "with_positions_offsets"\n },\n "sourceField" : {\n "name" : "_source",\n "enabled" : true\n },\n "properties" : {\n "id" : {\n "type" : "long",\n "indexName" : "id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "ancestor_ids" : {\n "type" : "long",\n "indexName" : "ancestor_ids",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "title" : {\n "type" : "string",\n "indexName" : "title",\n "index" : "analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "last_modified" : {\n "type" : "date",\n "indexName" : "last_modified",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "created" : {\n "type" : "date",\n "indexName" : "created",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "status" : {\n "type" : "string",\n "indexName" : "status",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "creator_id" : {\n "type" : "long",\n "indexName" : "creator_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "nullValue" : 0\n },\n "parent_id" : {\n "type" : "long",\n "indexName" : "parent_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n }\n }\n }\n}"
},
"notice" : {
"source" : "{\n "notice" : {\n "type" : "object",\n "dynamic" : true,\n "enabled" : true,\n "pathType" : "full",\n "dateFormats" : [ "dateOptionalTime" ],\n "boostField" : {\n "name" : "_boost"\n },\n "allField" : {\n "enabled" : true,\n "store" : "yes",\n "termVector" : "with_positions_offsets"\n },\n "sourceField" : {\n "name" : "_source",\n "enabled" : true\n },\n "properties" : {\n "has_name" : {\n "type" : "boolean",\n "indexName" : "has_name",\n "index" : "analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : false\n },\n "ancestor_ids" : {\n "type" : "long",\n "indexName" : "ancestor_ids",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "text" : {\n "type" : "string",\n "indexName" : "text",\n "index" : "analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "remote_last_modified" : {\n "type" : "date",\n "indexName" : "remote_last_modified",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "publish_date" : {\n "type" : "date",\n "indexName" : "publish_date",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "status" : {\n "type" : "string",\n "indexName" : "status",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "notice_type" : {\n "type" : "string",\n "indexName" : "notice_type",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "featured" : {\n "type" : "boolean",\n "indexName" : "featured",\n "index" : "analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : false\n },\n "creator_id" : {\n "type" : "long",\n "indexName" : "creator_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "nullValue" : 0\n },\n "id" : {\n "type" : "long",\n "indexName" : "id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "last_modified" : {\n "type" : "date",\n "indexName" : "last_modified",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "location_ids" : {\n "type" : "long",\n "indexName" : "location_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "created" : {\n "type" : "date",\n "indexName" : "created",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "name" : {\n "type" : "string",\n "indexName" : "name",\n "index" : "analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.2,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "sub_type" : {\n "type" : "string",\n "indexName" : "sub_type",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "parent_id" : {\n "type" : "long",\n "indexName" : "parent_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n }\n }\n }\n}"
},
"group" : {
"source" : "{\n "group" : {\n "type" : "object",\n "dynamic" : true,\n "enabled" : true,\n "pathType" : "full",\n "dateFormats" : [ "dateOptionalTime" ],\n "boostField" : {\n "name" : "_boost"\n },\n "allField" : {\n "enabled" : true,\n "store" : "yes",\n "termVector" : "with_positions_offsets"\n },\n "sourceField" : {\n "name" : "_source",\n "enabled" : true\n },\n "properties" : {\n "id" : {\n "type" : "long",\n "indexName" : "id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "ancestor_ids" : {\n "type" : "long",\n "indexName" : "ancestor_ids",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "last_modified" : {\n "type" : "date",\n "indexName" : "last_modified",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "created" : {\n "type" : "date",\n "indexName" : "created",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "status" : {\n "type" : "string",\n "indexName" : "status",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "name" : {\n "type" : "string",\n "indexName" : "name",\n "index" : "analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "text_id" : {\n "type" : "string",\n "indexName" : "text_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "creator_id" : {\n "type" : "long",\n "indexName" : "creator_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "nullValue" : 0\n },\n "parent_id" : {\n "type" : "long",\n "indexName" : "parent_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n }\n }\n }\n}"
},
"user" : {
"source" : "{\n "user" : {\n "type" : "object",\n "dynamic" : true,\n "enabled" : true,\n "pathType" : "full",\n "dateFormats" : [ "dateOptionalTime" ],\n "boostField" : {\n "name" : "_boost"\n },\n "allField" : {\n "enabled" : true,\n "store" : "yes",\n "termVector" : "with_positions_offsets"\n },\n "sourceField" : {\n "name" : "_source",\n "enabled" : true\n },\n "properties" : {\n "id" : {\n "type" : "long",\n "indexName" : "id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "ancestor_ids" : {\n "type" : "long",\n "indexName" : "ancestor_ids",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n },\n "username" : {\n "type" : "string",\n "indexName" : "username",\n "index" : "analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "last_modified" : {\n "type" : "date",\n "indexName" : "last_modified",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "email" : {\n "type" : "string",\n "indexName" : "email",\n "index" : "analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "created" : {\n "type" : "date",\n "indexName" : "created",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "format" : "yyyy-MM-dd HH:mm:ss"\n },\n "status" : {\n "type" : "string",\n "indexName" : "status",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "name" : {\n "type" : "string",\n "indexName" : "name",\n "index" : "analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : false,\n "omitTermFreqAndPositions" : false\n },\n "creator_id" : {\n "type" : "long",\n "indexName" : "creator_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4,\n "nullValue" : 0\n },\n "parent_id" : {\n "type" : "long",\n "indexName" : "parent_id",\n "index" : "not_analyzed",\n "store" : "no",\n "termVector" : "no",\n "boost" : 1.0,\n "omitNorms" : true,\n "omitTermFreqAndPositions" : true,\n "precisionStep" : 4\n }\n }\n }\n}"
}
}
}
}
}
}

SERVER 1:
[14:24:37,094][INFO ][server ] [Red Skull] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:10}: Initializing ...
[14:24:37,101][INFO ][plugins ] [Red Skull] Loaded []
[14:24:42,197][INFO ][server ] [Red Skull] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:10}: Initialized
[14:24:42,197][INFO ][server ] [Red Skull] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:10}: Starting ...
[14:24:42,585][INFO ][transport ] [Red Skull] boundAddress [inet[/10.224.103.159:9300]], publishAddress [inet[/10.224.103.159:9300]]
[14:24:45,861][INFO ][cluster.service ] [Red Skull] New Master [Red Skull][db1-13080][data][inet[/10.224.103.159:9300]]
[14:24:45,976][INFO ][discovery ] [Red Skull] iAnnounce/db1-13080
[14:24:46,030][INFO ][http ] [Red Skull] boundAddress [inet[/10.224.103.159:9200]], publishAddress [inet[/10.224.103.159:9200]]
[14:24:46,040][INFO ][cluster.metadata ] [Red Skull] Creating Index [ia_object_1270054545], shards [5]/[2], mappings [site, source, scraper, location, image, notice, group, user]
[14:24:46,664][INFO ][jmx ] [Red Skull] boundAddress [service:jmx:rmi:///jndi/rmi://:9400/jmxrmi], publishAddress [service:jmx:rmi:///jndi/rmi://10.224.103.159:9400/jmxrmi]
[14:24:46,664][INFO ][server ] [Red Skull] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:10}: Started
[14:25:06,637][WARN ][jgroups.pbcast.GMS ] db1-13080: failed to collect all ACKs (expected=1) for view [db1-13080|1] [db1-13080, db2-14372] after 2000ms, missing ACKs from [db1-13080]
[14:25:10,891][WARN ][jgroups.pbcast.GMS ] db1-13080: failed to collect all ACKs (expected=1) for unicast view [db1-13080|1] [db1-13080, db2-14372] after 2000ms, missing ACKs from [db2-14372]
[14:25:09,943][WARN ][jgroups.pbcast.NAKACK ] db1-13080: dropped message from db3-20667 (not in xmit_table), keys are [db1-13080, db2-14372], view=[db1-13080|1] [db1-13080, db2-14372]
[14:25:33,126][WARN ][jgroups.pbcast.NAKACK ] db1-13080: dropped message from db3-20667 (not in xmit_table), keys are [db1-13080, db2-14372], view=[db1-13080|1] [db1-13080, db2-14372]
[14:25:38,066][WARN ][jgroups.pbcast.NAKACK ] db1-13080: dropped message from db3-20667 (not in xmit_table), keys are [db1-13080], view=[db1-13080|2] [db1-13080]
[14:25:38,067][WARN ][jgroups.pbcast.GMS ] db1-13080: failed to collect all ACKs (expected=1) for view [db1-13080|2] [db1-13080] after 2000ms, missing ACKs from [db1-13080]
[14:25:58,253][WARN ][jgroups.pbcast.NAKACK ] db1-13080: dropped message from db3-20667 (not in xmit_table), keys are [db1-13080], view=[db1-13080|2] [db1-13080]
[14:25:58,254][WARN ][jgroups.pbcast.NAKACK ] db1-13080: dropped message from db3-20667 (not in xmit_table), keys are [db1-13080], view=[db1-13080|2] [db1-13080]
[14:26:07,173][WARN ][jgroups.pbcast.NAKACK ] db1-13080: dropped message from db3-20667 (not in xmit_table), keys are [db1-13080], view=[db1-13080|2] [db1-13080]
[14:26:11,311][INFO ][cluster.service ] [Red Skull] Added {[][org.elasticsearch.util.transport.DummyTransportAddress@307b37df],}
[14:26:11,539][WARN ][transport.netty ] [Red Skull] Failed to connect to discovered node [[][org.elasticsearch.util.transport.DummyTransportAddress@307b37df]]
java.lang.ClassCastException: org.elasticsearch.util.transport.DummyTransportAddress cannot be cast to org.elasticsearch.util.transport.InetSocketTransportAddress
at org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:437)
at org.elasticsearch.transport.netty.NettyTransport.nodesAdded(NettyTransport.java:396)
at org.elasticsearch.transport.TransportService.nodesAdded(TransportService.java:96)
at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:185)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
[14:26:11,766][WARN ][jgroups.pbcast.GMS ] db1-13080: failed to collect all ACKs (expected=1) for view MergeView::[db3-20667|3] [db3-20667, db1-13080], subgroups=[[db3-20667|2] [db3-20667], [db1-13080|2] [db1-13080]] after 2000ms, missing ACKs from [db1-13080]
[14:26:39,751][INFO ][cluster.service ] [Red Skull] Removed {[][org.elasticsearch.util.transport.DummyTransportAddress@307b37df],}
[14:26:44,513][INFO ][server ] [Red Skull] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:10}: Stopping ...
[14:29:39,097][INFO ][server ] [Red Skull] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:10}: Stopped
[14:29:39,098][INFO ][server ] [Red Skull] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:10}: Closing ...
[14:29:39,161][INFO ][server ] [Red Skull] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:10}: Closed
[14:33:44,049][INFO ][server ] [Blindspot] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:10}: Initializing ...
[14:33:44,054][INFO ][plugins ] [Blindspot] Loaded []
[14:33:46,857][INFO ][server ] [Blindspot] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:10}: Initialized
[14:33:46,858][INFO ][server ] [Blindspot] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:10}: Starting ...
[14:33:47,162][INFO ][transport ] [Blindspot] boundAddress [inet[/10.224.103.159:9300]], publishAddress [inet[/10.224.103.159:9300]]
[14:33:50,257][INFO ][cluster.service ] [Blindspot] New Master [Blindspot][db1-32838][data][inet[/10.224.103.159:9300]]
[14:33:50,321][INFO ][discovery ] [Blindspot] iAnnounce/db1-32838
[14:33:50,378][INFO ][http ] [Blindspot] boundAddress [inet[/10.224.103.159:9200]], publishAddress [inet[/10.224.103.159:9200]]
[14:33:50,595][INFO ][jmx ] [Blindspot] boundAddress [service:jmx:rmi:///jndi/rmi://:9400/jmxrmi], publishAddress [service:jmx:rmi:///jndi/rmi://10.224.103.159:9400/jmxrmi]
[14:33:50,595][INFO ][server ] [Blindspot] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:10}: Started
[14:35:00,834][INFO ][cluster.metadata ] [Blindspot] Creating Index [ia_object_1270054545], shards [5]/[2], mappings [site, source, scraper, location, image, notice, group, user]
[14:35:03,995][INFO ][cluster.service ] [Blindspot] Added {[Chaka][db2-22084][data][inet[ip-10-224-103-48.eu-west-1.compute.internal/10.224.103.48:9300]],}
[14:35:42,912][WARN ][jgroups.pbcast.GMS ] db3-28837 already present; returning existing view [db1-32838|2] [db1-32838, db2-22084, db3-28837]
[14:35:42,913][INFO ][cluster.service ] [Blindspot] Added {[Death's Head I&II][db3-28837][data][inet[ip-10-224-118-127.eu-west-1.compute.internal/10.224.118.127:9300]],}

SERVER 2:
[14:24:52,318][INFO ][server ] [Ghost Dancer] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:13}: Initializing ...
[14:24:52,323][INFO ][plugins ] [Ghost Dancer] Loaded []
[14:24:55,746][INFO ][server ] [Ghost Dancer] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:13}: Initialized
[14:24:55,746][INFO ][server ] [Ghost Dancer] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:13}: Starting ...
[14:24:56,069][INFO ][transport ] [Ghost Dancer] boundAddress [inet[/10.224.103.48:9300]], publishAddress [inet[/10.224.103.48:9300]]
[14:25:01,035][WARN ][jgroups.pbcast.GMS ] join(db2-14372) sent to db1-13080 timed out (after 3000 ms), retrying
[14:25:01,071][INFO ][cluster.service ] [Ghost Dancer] New Master [Ghost Dancer][db2-14372][data][inet[/10.224.103.48:9300]]
[14:25:01,133][INFO ][discovery ] [Ghost Dancer] iAnnounce/db2-14372
[14:25:01,163][INFO ][http ] [Ghost Dancer] boundAddress [inet[/10.224.103.48:9200]], publishAddress [inet[/10.224.103.48:9200]]
[14:25:01,532][INFO ][jmx ] [Ghost Dancer] boundAddress [service:jmx:rmi:///jndi/rmi://:9400/jmxrmi], publishAddress [service:jmx:rmi:///jndi/rmi://10.224.103.48:9400/jmxrmi]
[14:25:01,532][INFO ][server ] [Ghost Dancer] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:13}: Started
[14:25:01,619][INFO ][cluster.metadata ] [Ghost Dancer] Creating Index [ia_object_1270054545], shards [5]/[2], mappings [site, scraper, source, location, image, group, notice, user]
[14:25:09,791][WARN ][jgroups.pbcast.NAKACK ] db2-14372: dropped message from db1-13080 (not in xmit_table), keys are [db3-20667, db2-14372], view=[db2-14372|1] [db2-14372, db3-20667]
[14:25:16,272][WARN ][jgroups.pbcast.NAKACK ](requester=db1-13080, local_addr=db2-14372) message db2-14372::1 not found in retransmission table of db2-14372:
[3 : 6 (6) (size=3, missing=0, highest stability=3)]
[14:25:33,962][WARN ][jgroups.pbcast.NAKACK ](requester=db1-13080, local_addr=db2-14372) message db2-14372::2 not found in retransmission table of db2-14372:
[3 : 6 (6) (size=3, missing=0, highest stability=3)]
[14:25:33,962][WARN ][jgroups.pbcast.NAKACK ](requester=db1-13080, local_addr=db2-14372) message db2-14372::3 not found in retransmission table of db2-14372:
[3 : 6 (6) (size=3, missing=0, highest stability=3)]
[14:25:34,257][WARN ][jgroups.pbcast.NAKACK ](requester=db1-13080, local_addr=db2-14372) message db2-14372::1 not found in retransmission table of db2-14372:
[3 : 6 (6) (size=3, missing=0, highest stability=3)]
[14:25:34,257][WARN ][jgroups.pbcast.NAKACK ](requester=db1-13080, local_addr=db2-14372) message db2-14372::2 not found in retransmission table of db2-14372:
[3 : 6 (6) (size=3, missing=0, highest stability=3)]
[14:25:34,257][WARN ][jgroups.pbcast.NAKACK ](requester=db1-13080, local_addr=db2-14372) message db2-14372::3 not found in retransmission table of db2-14372:
[3 : 6 (6) (size=3, missing=0, highest stability=3)]
[14:25:41,850][WARN ][jgroups.pbcast.NAKACK ](requester=db1-13080, local_addr=db2-14372) message db2-14372::1 not found in retransmission table of db2-14372:
[3 : 6 (6) (size=3, missing=0, highest stability=3)]
[14:25:41,850][WARN ][jgroups.pbcast.NAKACK ](requester=db1-13080, local_addr=db2-14372) message db2-14372::2 not found in retransmission table of db2-14372:
[3 : 6 (6) (size=3, missing=0, highest stability=3)]
[14:25:41,850][WARN ][jgroups.pbcast.NAKACK ](requester=db1-13080, local_addr=db2-14372) message db2-14372::3 not found in retransmission table of db2-14372:
[3 : 6 (6) (size=3, missing=0, highest stability=3)]
[14:26:33,856][INFO ][cluster.service ] [Ghost Dancer] Added {[War Machine][db3-20667][data][inet[ip-10-224-118-127.eu-west-1.compute.internal/10.224.118.127:9300]],}
[14:26:42,309][WARN ][jgroups.pbcast.NAKACK ] db2-14372: dropped message from db1-13080 (not in xmit_table), keys are [db3-20667, db2-14372], view=[db2-14372|1] [db2-14372, db3-20667]
[14:27:04,703][WARN ][jgroups.pbcast.NAKACK ] db2-14372: dropped message from db1-13080 (not in xmit_table), keys are [db3-20667, db2-14372], view=[db2-14372|1] [db2-14372, db3-20667]
[14:27:04,703][WARN ][jgroups.pbcast.NAKACK ] db2-14372: dropped message from db1-13080 (not in xmit_table), keys are [db3-20667, db2-14372], view=[db2-14372|1] [db2-14372, db3-20667]
[14:27:14,552][WARN ][jgroups.pbcast.NAKACK ] db2-14372: dropped message from db1-13080 (not in xmit_table), keys are [db3-20667, db2-14372], view=[db2-14372|1] [db2-14372, db3-20667]
[14:27:15,125][WARN ][jgroups.pbcast.NAKACK ] db2-14372: dropped message from db1-13080 (not in xmit_table), keys are [db3-20667, db2-14372], view=[db2-14372|1] [db2-14372, db3-20667]
[14:27:33,301][INFO ][server ] [Ghost Dancer] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:13}: Stopping ...
[14:28:04,529][WARN ][jgroups.pbcast.NAKACK ] db2-14372: dropped message from db1-13080 (not in xmit_table), keys are [db3-20667, db2-14372], view=[db2-14372|1] [db2-14372, db3-20667]
[14:28:12,960][WARN ][jgroups.pbcast.GMS ] db2-14372: failed to collect all ACKs (expected=1) for view [db3-20667|2] [db3-20667] after 2000ms, missing ACKs from [db3-20667]
[14:28:20,577][ERROR][jgroups.UNICAST ] db2-14372: sender window for db2-14372 not found
[14:31:25,936][INFO ][server ] [Ghost Dancer] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:13}: Stopped
[14:31:25,937][INFO ][server ] [Ghost Dancer] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:13}: Closing ...
[14:31:25,955][INFO ][server ] [Ghost Dancer] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:13}: Closed
[14:34:57,406][INFO ][server ] [Chaka] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:13}: Initializing ...
[14:34:57,411][INFO ][plugins ] [Chaka] Loaded []
[14:35:00,373][INFO ][server ] [Chaka] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:13}: Initialized
[14:35:00,374][INFO ][server ] [Chaka] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:13}: Starting ...
[14:35:00,703][INFO ][transport ] [Chaka] boundAddress [inet[/10.224.103.48:9300]], publishAddress [inet[/10.224.103.48:9300]]
[14:35:15,873][INFO ][cluster.service ] [Chaka] Detected Master [Blindspot][db1-32838][data][inet[ip-10-224-103-159.eu-west-1.compute.internal/10.224.103.159:9300]], Added {[Blindspot][db1-32838][data][inet[ip-10-224-103-159.eu-west-1.compute.internal/10.224.103.159:9300]],}
[14:35:16,774][INFO ][discovery ] [Chaka] iAnnounce/db2-22084
[14:35:16,801][INFO ][http ] [Chaka] boundAddress [inet[/10.224.103.48:9200]], publishAddress [inet[/10.224.103.48:9200]]
[14:35:16,886][INFO ][jmx ] [Chaka] boundAddress [service:jmx:rmi:///jndi/rmi://:9400/jmxrmi], publishAddress [service:jmx:rmi:///jndi/rmi://10.224.103.48:9400/jmxrmi]
[14:35:16,887][INFO ][server ] [Chaka] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:13}: Started
[14:35:44,296][INFO ][cluster.service ] [Chaka] Added {[Death's Head I&II][db3-28837][data][inet[ip-10-224-118-127.eu-west-1.compute.internal/10.224.118.127:9300]],}

SERVER 3:
[14:24:55,498][INFO ][server ] [War Machine] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:22}: Initializing ...
[14:24:55,515][INFO ][plugins ] [War Machine] Loaded []
[14:24:58,908][INFO ][server ] [War Machine] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:22}: Initialized
[14:24:58,908][INFO ][server ] [War Machine] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:22}: Starting ...
[14:24:59,179][INFO ][transport ] [War Machine] boundAddress [inet[/10.224.118.127:9300]], publishAddress [inet[/10.224.118.127:9300]]
[14:25:08,919][WARN ][jgroups.pbcast.NAKACK ] db3-20667: dropped message from db1-13080 (not in xmit_table), keys are [db3-20667, db2-14372], view=[db2-14372|1] [db2-14372, db3-20667]
[14:25:31,512][WARN ][discovery ] [War Machine] Waited for 30s and no initial state was set by the discovery
[14:25:31,512][INFO ][discovery ] [War Machine] iAnnounce/db3-20667
[14:25:31,524][INFO ][http ] [War Machine] boundAddress [inet[/10.224.118.127:9200]], publishAddress [inet[/10.224.118.127:9200]]
[14:25:31,845][INFO ][jmx ] [War Machine] boundAddress [service:jmx:rmi:///jndi/rmi://:9400/jmxrmi], publishAddress [service:jmx:rmi:///jndi/rmi://10.224.118.127:9400/jmxrmi]
[14:25:31,845][INFO ][server ] [War Machine] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:22}: Started
[14:25:33,407][INFO ][cluster.service ] [War Machine] New Master [War Machine][db3-20667][data][inet[ip-10-224-118-127.eu-west-1.compute.internal/10.224.118.127:9300]]
[14:25:35,048][WARN ][jgroups.pbcast.NAKACK ] db3-20667: dropped message from db1-13080 (not in xmit_table), keys are [db3-20667], view=[db3-20667|2] [db3-20667]
[14:25:57,708][WARN ][jgroups.pbcast.NAKACK ] db3-20667: dropped message from db1-13080 (not in xmit_table), keys are [db3-20667], view=[db3-20667|2] [db3-20667]
[14:25:59,258][WARN ][jgroups.pbcast.NAKACK ] db3-20667: dropped message from db1-13080 (not in xmit_table), keys are [db3-20667], view=[db3-20667|2] [db3-20667]
[14:26:07,242][WARN ][jgroups.pbcast.NAKACK ] db3-20667: dropped message from db1-13080 (not in xmit_table), keys are [db3-20667], view=[db3-20667|2] [db3-20667]
[14:26:29,259][WARN ][jgroups.blocks.TCPConnectionMap] Could not read accept connection from peer java.net.SocketTimeoutException: Read timed out
[14:26:36,756][INFO ][server ] [War Machine] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:22}: Stopping ...
[14:26:37,382][INFO ][server ] [War Machine] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:22}: Stopped
[14:26:37,382][INFO ][server ] [War Machine] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:22}: Closing ...
[14:26:37,387][INFO ][server ] [War Machine] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:22}: Closed
[14:35:33,336][INFO ][server ] [Death's Head I&II] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:22}: Initializing ...
[14:35:33,349][INFO ][plugins ] [Death's Head I&II] Loaded []
[14:35:35,510][INFO ][server ] [Death's Head I&II] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:22}: Initialized
[14:35:35,510][INFO ][server ] [Death's Head I&II] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:22}: Starting ...
[14:35:35,729][INFO ][transport ] [Death's Head I&II] boundAddress [inet[/10.224.118.127:9300]], publishAddress [inet[/10.224.118.127:9300]]
[14:35:38,813][WARN ][jgroups.pbcast.GMS ] join(db3-28837) sent to db1-32838 timed out (after 3000 ms), retrying
[14:35:41,817][WARN ][jgroups.pbcast.GMS ] join(db3-28837) sent to db1-32838 timed out (after 3000 ms), retrying
[14:35:44,056][INFO ][cluster.service ] [Death's Head I&II] Detected Master [Blindspot][db1-32838][data][inet[ip-10-224-103-159.eu-west-1.compute.internal/10.224.103.159:9300]], Added {[Chaka][db2-22084][data][inet[ip-10-224-103-48.eu-west-1.compute.internal/10.224.103.48:9300]],[Blindspot][db1-32838][data][inet[ip-10-224-103-159.eu-west-1.compute.internal/10.224.103.159:9300]],}
[14:35:44,907][INFO ][discovery ] [Death's Head I&II] iAnnounce/db3-28837
[14:35:44,932][INFO ][http ] [Death's Head I&II] boundAddress [inet[/10.224.118.127:9200]], publishAddress [inet[/10.224.118.127:9200]]
[14:35:45,004][INFO ][jmx ] [Death's Head I&II] boundAddress [service:jmx:rmi:///jndi/rmi://:9400/jmxrmi], publishAddress [service:jmx:rmi:///jndi/rmi://10.224.118.127:9400/jmxrmi]
[14:35:45,004][INFO ][server ] [Death's Head I&II] {ElasticSearch/0.6.0-SNAPSHOT/2010-03-31T14:31:22}: Started
@kimchy
Copy link
Member

kimchy commented Apr 1, 2010

Hey,

Its a bit hard to nail this since you descrive two problems that I think are unrelated. It would be great to nail down one at a time.

First, what is your jgroups discovery module configuration?

Second, regarding the single node starting and not recovering from the gateway, can you please set the gateway.fs logging to TRACE in the logging.yml file and post the log of that server (the part where it starts up and does not read it)?

Third, the other problems, of nodes not discovering each other. Thats a bit of a nasty one. Lets see what your jgroups configuration is and continue from there...

@clintongormley
Copy link
Contributor Author

My jgroups config:

network:
bindHost: 10.224.103.159
discovery:
jgroups:
config: tcp
tcpping:
initial_hosts: 10.224.103.159[7800],10.224.103.48[7800],10.224.118.127[7800]

I set my logging to:
rootLogger: INFO, console, file
logger:
jgroups: WARN

gateway:
     fs:      TRACE

Or should that be this instead?
gateway.fs: TRACE

Btw, the issue of not seeing the index until two nodes have started only seems to happen after I've had a bad shutdown eg couldn't snapshot or something like that

@kimchy
Copy link
Member

kimchy commented Apr 1, 2010

Configuration looks good. Regarding the logging, both option work, but note, this only applies to the single node starting and not recovering from the gateway.

Regarding the 3 notes started at the same time, I just pushed a more verbose logging into the jgroups discovery module, which I hope will help to nail down that problem. Set the discovery.jgroups to DEBUG and send me the 3 servers logs.

Lets try and nail these problems separately, otherwise I will get confused ;)

@kimchy
Copy link
Member

kimchy commented May 28, 2010

Are we ok with this one, can I close this?

@clintongormley
Copy link
Contributor Author

Yes - I think so. In retrospect, the problem was probably timeouts caused by the unthrottled recovery

@bluelu bluelu mentioned this issue Dec 11, 2014
Closed
@makeyang makeyang mentioned this issue Sep 7, 2015
Closed
rmuir pushed a commit to rmuir/elasticsearch that referenced this issue Nov 8, 2015
@dadoonet
[Test] Use now full qualified names for fields
ec0de9c 
We were asking for short name fields but elasticsearch does not allow anymore using short names but full qualified names.

```java
SearchResponse response = client().prepareSearch("test")
        .addField("content_type")
        .addField("name")
        .execute().get();
```

We need to use now:

```java
SearchResponse response = client().prepareSearch("test")
        .addField("file.content_type")
        .addField("file.name")
        .execute().get();
```

Closes elastic#102.
@jfenc91 jfenc91 mentioned this issue Jun 1, 2016
Closed
C-Bish pushed a commit to C-Bish/elasticsearch that referenced this issue Mar 22, 2017
@kblincoe
Merge pull request elastic#102 from fernandozhu/master
dc4e6cb 
Claim issue 23673
ClaudioMFreitas pushed a commit to ClaudioMFreitas/elasticsearch-1 that referenced this issue Nov 12, 2019
Merge pull request elastic#102 from CyVerse-Ansible/task-tags
03f4a8f 
Allow for selective command line task execution
@Happyfeet01 Happyfeet01 mentioned this issue May 24, 2020
Closed
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kimchy @clintongormley