[CI] MixedClusterClientYamlTestSuiteIT test {p0=search/110_field_collapsing/field collapsing and inner_hits} #52416
Assignees
Labels
:Search/Search
Search-related issues that do not fall into other categories
Team:Search
Meta label for search team
>test-failure
Triaged test failures from CI
Comments
astefan
added
:Search/Search
|
Pinging @elastic/es-search (:Search/Search) |
|
It looks like this test was run against 6.8 mixed cluster: This doesn't reproduce for me. It doesn't reproduce for me just on 7.x cluster either: The stashed output from the failed test is: "stash" : {
"body" : {
"took" : 12,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 5,
"max_score" : null,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "6",
"_score" : null,
"_source" : {
"sort" : 36,
"numeric_group" : 3
},
"fields" : {
"numeric_group" : [
3
]
},
"sort" : [
36
],
"inner_hits" : {
"sub_hits" : {
"hits" : {
"total" : 1,
"max_score" : null,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "6",
"_score" : null,
"_source" : {
"sort" : 36,
"numeric_group" : 3
},
"sort" : [
36
]
}
]
}
}
}
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "3",
"_score" : null,
"_source" : {
"sort" : 24,
"numeric_group" : 1
},
"fields" : {
"numeric_group" : [
1
]
},
"sort" : [
24
],
"inner_hits" : {
"sub_hits" : {
"hits" : {
"total" : 3,
"max_score" : null,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "2",
"_score" : null,
"_source" : {
"sort" : 6,
"numeric_group" : 1
},
"sort" : [
6
]
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "1",
"_score" : "-Infinity",
"_source" : {
"sort" : 10,
"numeric_group" : 1
},
"sort" : [
10
]
}
]
}
}
}
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "4",
"_score" : null,
"_source" : {
"sort" : 10,
"numeric_group" : 25
},
"fields" : {
"numeric_group" : [
25
]
},
"sort" : [
10
],
"inner_hits" : {
"sub_hits" : {
"hits" : {
"total" : 2,
"max_score" : null,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "5",
"_score" : "-Infinity",
"_source" : {
"sort" : 5,
"numeric_group" : 25
},
"sort" : [
5
]
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "4",
"_score" : null,
"_source" : {
"sort" : 10,
"numeric_group" : 25
},
"sort" : [
10
]
}
]
}
}
}
}
]
}
}
}
}]As expected, 3 collapsed groups returned with hits.total: 1, 3 and 2 which should sum up to 6, but incorrectly summed up to 5. |
|
There were no failures of this test after Feb 17. There is another failure of this test from Jan 27 but on different condition: java.lang.AssertionError: hits.hits.1.inner_hits.sub_hits.hits.total didn't match expected value: | Here in the missing hit in the second group for inner hits, a doc with "_id" : "2", "sort" : 6 . [{
"stash" : {
"body" : {
"took" : 32,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 6,
"max_score" : null,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "6",
"_score" : "-Infinity",
"_source" : {
"sort" : 36,
"numeric_group" : 3
},
"fields" : {
"numeric_group" : [
3
]
},
"sort" : [
36
],
"inner_hits" : {
"sub_hits" : {
"hits" : {
"total" : 1,
"max_score" : null,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "6",
"_score" : "-Infinity",
"_source" : {
"sort" : 36,
"numeric_group" : 3
},
"sort" : [
36
]
}
]
}
}
}
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "3",
"_score" : null,
"_source" : {
"sort" : 24,
"numeric_group" : 1
},
"fields" : {
"numeric_group" : [
1
]
},
"sort" : [
24
],
"inner_hits" : {
"sub_hits" : {
"hits" : {
"total" : 2,
"max_score" : null,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "2",
"_score" : "-Infinity",
"_source" : {
"sort" : 6,
"numeric_group" : 1
},
"sort" : [
6
]
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "3",
"_score" : null,
"_source" : {
"sort" : 24,
"numeric_group" : 1
},
"sort" : [
24
]
}
]
}
}
}
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "4",
"_score" : "-Infinity",
"_source" : {
"sort" : 10,
"numeric_group" : 25
},
"fields" : {
"numeric_group" : [
25
]
},
"sort" : [
10
],
"inner_hits" : {
"sub_hits" : {
"hits" : {
"total" : 2,
"max_score" : null,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "5",
"_score" : null,
"_source" : {
"sort" : 5,
"numeric_group" : 25
},
"sort" : [
5
]
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "4",
"_score" : "-Infinity",
"_source" : {
"sort" : 10,
"numeric_group" : 25
},
"sort" : [
10
]
}
]
}
}
}
}
]
}
}
}
}] |
|
Another failure from Nov 1, 2019 Here there is also a missing document in the second group [{
"stash" : {
"body" : {
"took" : 18,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 5,
"max_score" : null,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "6",
"_score" : null,
"_source" : {
"sort" : 36,
"numeric_group" : 3
},
"fields" : {
"numeric_group" : [
3
]
},
"sort" : [
36
],
"inner_hits" : {
"sub_hits" : {
"hits" : {
"total" : 1,
"max_score" : null,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "6",
"_score" : null,
"_source" : {
"sort" : 36,
"numeric_group" : 3
},
"sort" : [
36
]
}
]
}
}
}
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "1",
"_score" : null,
"_source" : {
"sort" : 10,
"numeric_group" : 1
},
"fields" : {
"numeric_group" : [
1
]
},
"sort" : [
10
],
"inner_hits" : {
"sub_hits" : {
"hits" : {
"total" : 3,
"max_score" : null,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "2",
"_score" : null,
"_source" : {
"sort" : 6,
"numeric_group" : 1
},
"sort" : [
6
]
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "1",
"_score" : null,
"_source" : {
"sort" : 10,
"numeric_group" : 1
},
"sort" : [
10
]
}
]
}
}
}
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "4",
"_score" : null,
"_source" : {
"sort" : 10,
"numeric_group" : 25
},
"fields" : {
"numeric_group" : [
25
]
},
"sort" : [
10
],
"inner_hits" : {
"sub_hits" : {
"hits" : {
"total" : 2,
"max_score" : null,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "5",
"_score" : null,
"_source" : {
"sort" : 5,
"numeric_group" : 25
},
"sort" : [
5
]
},
{
"_index" : "test",
"_type" : "_doc",
"_id" : "4",
"_score" : null,
"_source" : {
"sort" : 10,
"numeric_group" : 25
},
"sort" : [
10
]
}
]
}
}
}
}
]
}
}
}
}] |
|
it looks like some docs were missed during collection in some cases during the initial query phase, or in other cases during inner_hits phase. I am out of ideas how this could happen. |
|
Another failure on Apr 3 on 7.x windows. Build scan: https://gradle-enterprise.elastic.co/s/zh4vk2ddkfdc2 java.lang.AssertionError: Failure at [search/110_field_collapsing:67]: hits.total didn't match expected value: [{
1> "stash" : {
1> "body" : {
1> "took" : 18,
1> "timed_out" : false,
1> "_shards" : {
1> "total" : 5,
1> "successful" : 5,
1> "skipped" : 0,
1> "failed" : 0
1> },
1> "hits" : {
1> "total" : 3,
1> "max_score" : null,
1> "hits" : [
1> {
1> "_index" : "test",
1> "_type" : "_doc",
1> "_id" : "3",
1> "_score" : null,
1> "_source" : {
1> "sort" : 24,
1> "numeric_group" : 1
1> },
1> "fields" : {
1> "numeric_group" : [
1> 1
1> ]
1> },
1> "sort" : [
1> 24
1> ]
1> },
1> {
1> "_index" : "test",
1> "_type" : "_doc",
1> "_id" : "5",
1> "_score" : "-Infinity",
1> "_source" : {
1> "sort" : 5,
1> "numeric_group" : 25
1> },
1> "fields" : {
1> "numeric_group" : [
1> 25
1> ]
1> },
1> "sort" : [
1> 5
1> ]
1> }
1> ]
1> }
1> }
1> }
1> }] |
mayya-sharipova
added a commit
to mayya-sharipova/elasticsearch
that referenced
this issue
Apr 6, 2020
There were some failures on 7.x of field collapse tests, where total hits count was less then expected. This adds an additional test to check total hits count before field collapse queries to understand if the problem is with field collapsing or with simply that writes have not been finished yet Relates to elastic#52416
mayya-sharipova
added a commit
that referenced
this issue
Apr 6, 2020
There were some failures on 7.x of field collapse tests, where total hits count was less then expected. This adds an additional test to check total hits count before field collapse queries to understand if the problem is with field collapsing or with simply that writes have not been finished yet Relates to #52416
|
Another instance of this failure just popped up: https://gradle-enterprise.elastic.co/s/xfqfl3g3q6th2 |
|
Yet another failure: https://gradle-enterprise.elastic.co/s/4tzbht3q5hqtm stack dump: |
|
Another failure: https://gradle-enterprise.elastic.co/s/3no2o4uqgdf7q same error: |
|
Another failure https://gradle-enterprise.elastic.co/s/sll6vajxn24qa |
|
Another failure https://gradle-enterprise.elastic.co/s/tn64v7lwgntzm |
|
Another failure in https://gradle-enterprise.elastic.co/s/qls5tmomhsvpi |
|
A similar failure, but for |
|
This looks also related but is in ' 110_field_collapsing/no hits and inner_hits': https://gradle-enterprise.elastic.co/s/yv4kj4xx7bqj2 |
|
another failure https://gradle-enterprise.elastic.co/s/moy4dga4g5pyc |
|
another one: https://gradle-enterprise.elastic.co/s/jpievlatqzaky |
|
Another failure on 7.10: https://gradle-enterprise.elastic.co/s/h5mb2lkr3witq: reproduction line (does not reproduce locally): There is also a related test failing with the reproduction line (does not reproduce locally): I'll mute these tests now on |
This was referenced Nov 11, 2020
danielmitterdorfer
added a commit
to danielmitterdorfer/elasticsearch
that referenced
this issue
Nov 11, 2020
danielmitterdorfer
added a commit
that referenced
this issue
Nov 11, 2020
|
Another one: https://gradle-enterprise.elastic.co/s/plgo6plomydn2 The exact line (339) where it failed is different from the above failures. But they are from the same test suite. |
|
Another failure: https://gradle-enterprise.elastic.co/s/t66afxe5e3nr4 |
|
Might be related to #66728 |
|
Exact same failure. Still a mixed cluster test against 6.8 and 7.x. https://gradle-enterprise.elastic.co/s/nboqmld334tyq |
|
another failure: https://gradle-enterprise.elastic.co/s/ea4dzwxzoo7aw |
|
Two other instances that happened today on 7.11 and 7.x branches: |
|
More today: |
DaveCTurner
added a commit
to javanna/elasticsearch
that referenced
this issue
Jan 31, 2021
|
Another related failure in 7.11: https://gradle-enterprise.elastic.co/s/4dv4mhb4skwna Muted in #69587 |
|
https://gradle-enterprise.elastic.co/s/mazkqci6eugzc Happened again but this time in a different test: |
mayya-sharipova
added a commit
to mayya-sharipova/elasticsearch
that referenced
this issue
Mar 1, 2021
Break up field collapsting tests into 2: 1) simple indexing and 2) indexingwith versions to check if a problem of documents missing could be because of versions. Relates to elastic#52416
mayya-sharipova
added a commit
that referenced
this issue
Mar 8, 2021
Break up field collapsting tests into 2: 1) simple indexing and 2) indexingwith versions to check if a problem of documents missing could be because of versions. Relates to #52416
|
Another related failure here https://gradle-enterprise.elastic.co/s/shre4iovbbwly |
The test that failed was |
|
Got another one! |
|
I will mute the test today. |
dnhatn
added a commit
that referenced
this issue
Mar 17, 2021
|
I think shard relocations are causing these failures. I've disabled relocations in this test in 0675b14. I will dig into it deeper later. |
dnhatn
added a commit
that referenced
this issue
Mar 18, 2021
dnhatn
added a commit
that referenced
this issue
Mar 18, 2021
|
I've removed AwaitsFix for this test. Please open a new issue if this test fails again. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
This one doesn't reproduce for me.
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-windows-compatibility/os=windows-2019/442/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-windows-compatibility/os=windows-2016/442/console
https://gradle-enterprise.elastic.co/s/7sb66cpgvp472
https://gradle-enterprise.elastic.co/s/ebbbgtn25jytw
The text was updated successfully, but these errors were encountered: