JSON error in snapshots: Duplicate field #28906
Comments
javanna
added
the
:Distributed Coordination/Snapshot/Restore
Some were created with 5.6.4. Do you think that reindexing them would help or should I just set |
|
I think that reindexing them will raise the error at index time, then you'll have to fix those docs in order to get them indexed. Setting the system property is a work-around, not really a solution. |
|
By the way, I think I should mention that the error is Where I don't know anything about the internal snapshot representation, but this seems to be a problem in the snapshot metadata, not in an index. (The snapshots were all created using 6.1.0) |
|
@JD557 Do you perhaps have multiple clusters that are concurrently writing to this repository? |
|
I have a single cluster with 3 nodes writing to that repository and a hourly curator job to create a snapshot. The curator job accesses the cluster via DNS, so the snapshot requests don't always hit the same machine (but, AFAIK, there are never 2 snapshots running at the same time). |
|
Are you experiencing frequent network partitions between the nodes? Anything in the logs on the days leading up to this that could indicate failures on some of the nodes? Looking at the following snapshotting code, we should make it more resilient against duplicates (we check for duplicates at snapshot initialization, but not when writing the I still don't understand though how we can end up in the presented situation, except for concurrent access to the repository. |
|
I don't think that there are frequent network partitions. However, looking at the logs, here are some errors that happened before the first time that I had this problem (sorry for the large logs): Snapshot creation failed Snapshot deletion failed Concurrent snapshot exceptions (Maybe a snapshot got "stuck" and took too much to complete?) And finally, I got a Maybe this can be triggered by transient failures in S3? |
|
Discussed with @tlrx: We are not sure how we can end up in this situation, but we plan on adding a check in |
|
Is this check implemented in 6.2.4? |
…#29634) Adds a check in BlobstoreRepository.snapshot(...) that prevents duplicate snapshot names and fails the snapshot before writing out the new index file. This ensures that you cannot end up in this situation where the index file has duplicate names and cannot be read anymore . Relates to #28906
…#29634) Adds a check in BlobstoreRepository.snapshot(...) that prevents duplicate snapshot names and fails the snapshot before writing out the new index file. This ensures that you cannot end up in this situation where the index file has duplicate names and cannot be read anymore . Relates to #28906
|
The fix will be in 6.3 |
|
Talked to @ywelsch today, now the check for duplicate snapshot name has been added we think that we can't really do much for now and we are both in favor of closing this issue. If it reappears on versions > 6.3 then we'll reopen for investigation. |
|
my issue is related, i think... type":"pcap_file","_id":null,"status":400,"error":{"type":"parse_exception","reason":"Failed to parse content to map","caused_by":{"type":"json_parse_exception","reason":"Duplicate field 'ip_ip_addr'\n at |
|
@tjc808 I don't think this is related to snapshot/restore feature but instead because you're indexing documents with duplicated fields. You can ask questions like these in the Elastic forums instead. |
|
The field name |
|
I am using org.elasticsearch.client:elasticsearch-rest-high-level-client version 6.8.23. I cannot use newer versions due to company requirements. com.fasterxml.jackson.core.JsonParseException: Duplicate field 'id' Here is a simplified JSON for IndexRequest that I hope reproduces the problem. { |
Elasticsearch version (
bin/elasticsearch --version): 6.1.0Plugins installed: [discovery-ec2, repository-s3, ingest-geoip, ingest-user-agent]
JVM version (
java -version): same asdocker.elastic.co/elasticsearch/elasticsearch-oss:6.1.0OS version (
uname -aif on a Unix-like system): same asdocker.elastic.co/elasticsearch/elasticsearch-oss:6.1.0Description of the problem including expected versus actual behavior:
After leaving the system running with automatic hourly snapshots to S3 for a few days, I get the following error:
I already created a new snapshot repository and deleted the old data. This fixed the problem for a while, but it appeared again.
The system in question has ~840 indices. Maybe the large number of indices makes this problem more common? (I haven't seen this on other systems).
The text was updated successfully, but these errors were encountered: