Skip to content

[CI][7.x] CCR downgrade-to-basic-license timeout waiting for yellow #52126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
tvernum opened this issue Feb 10, 2020 · 2 comments · Fixed by #52327
Closed

[CI][7.x] CCR downgrade-to-basic-license timeout waiting for yellow #52126

tvernum opened this issue Feb 10, 2020 · 2 comments · Fixed by #52327
Assignees
@dnhatn
Labels
:Distributed Indexing/CCR Issues around the Cross Cluster State Replication features >test-failure Triaged test failures from CI

Comments

@tvernum
Copy link
Contributor

tvernum commented Feb 10, 2020

Doesn't reproduce

./gradlew ':x-pack:plugin:ccr:qa:downgrade-to-basic-license:follow-clusterRunner' \
  --tests "org.elasticsearch.xpack.ccr.FollowIndexIT.testDowngradeRemoteClusterToBasic" \
  -Dtests.seed=F143CD0F19764898 \
  -Dtests.security.manager=true \
  -Dtests.locale=ar-IQ -Dtests.timezone=CET \
  -Dcompiler.java=13

rg.elasticsearch.xpack.ccr.FollowIndexIT > testDowngradeRemoteClusterToBasic FAILED	
    org.elasticsearch.client.ResponseException: method [GET], host [http://127.0.0.1:39119], URI [/_cluster/health/logs-20190101?level=shards&wait_for_no_initializing_shards=true&wait_for_status=yellow&wait_for_active_shards=1&wait_for_no_relocating_shards=true&timeout=5s], status line [HTTP/1.1 408 Request Timeout]	
    {"cluster_name":"follow-cluster","status":"yellow","timed_out":true,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":1,"unassigned_shards":1,"delayed_unassigned_shards":0,"number_of_pending_tasks":8,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":2647,"active_shards_percent_as_number":50.0,"indices":{"logs-20190101":{"status":"yellow","number_of_shards":1,"number_of_replicas":1,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":1,"unassigned_shards":1,"shards":{"0":{"status":"yellow","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":1,"unassigned_shards":1}}}}}	
        at __randomizedtesting.SeedInfo.seed([F143CD0F19764898:9DD8623DF648135E]:0)	
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283)	
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261)	
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)	
        at org.elasticsearch.xpack.ccr.ESCCRRestTestCase.ensureYellow(ESCCRRestTestCase.java:225)	
        at org.elasticsearch.xpack.ccr.ESCCRRestTestCase.ensureYellow(ESCCRRestTestCase.java:214)	
        at org.elasticsearch.xpack.ccr.FollowIndexIT.lambda$testDowngradeRemoteClusterToBasic$0(FollowIndexIT.java:49)	
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:893)	
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:878)	
        at org.elasticsearch.xpack.ccr.FollowIndexIT.testDowngradeRemoteClusterToBasic(FollowIndexIT.java:48)
Node Log

Follow Cluster Log

» [2020-02-10T04:42:21,815][DEBUG][o.e.c.c.C.CoordinatorPublication] [follow-cluster-0] publication ended successfully: Publication{term=1, version=16}	
» [2020-02-10T04:42:21,851][DEBUG][o.e.c.c.PublicationTransportHandler] [follow-cluster-0] received diff cluster state version [17] with uuid [mGX1C2WuSqGQ8pM16E3SdA], diff size [560]	
» [2020-02-10T04:42:22,055][INFO ][o.e.l.LicenseService     ] [follow-cluster-0] license [3c73c800-1bb0-4d06-b95a-a5899fa73228] mode [trial] - valid	
» [2020-02-10T04:42:22,056][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [follow-cluster-0] Active license is now [TRIAL]; Security is disabled	
» [2020-02-10T04:42:22,057][DEBUG][o.e.c.c.C.CoordinatorPublication] [follow-cluster-0] publication ended successfully: Publication{term=1, version=17}	
» [2020-02-10T04:42:22,072][DEBUG][o.e.c.c.PublicationTransportHandler] [follow-cluster-0] received diff cluster state version [18] with uuid [ACM7Z8_LQv6RonPKKWfpag], diff size [294]	
» [2020-02-10T04:42:22,181][DEBUG][o.e.c.c.C.CoordinatorPublication] [follow-cluster-0] publication ended successfully: Publication{term=1, version=18}	
» [2020-02-10T04:42:22,187][DEBUG][o.e.c.c.PublicationTransportHandler] [follow-cluster-0] received diff cluster state version [19] with uuid [QeGPtBzDShuxRGTVs4LBGQ], diff size [427]	
» [2020-02-10T04:42:22,332][DEBUG][o.e.c.c.C.CoordinatorPublication] [follow-cluster-0] publication ended successfully: Publication{term=1, version=19}	
» [2020-02-10T04:42:26,842][DEBUG][o.e.c.c.PublicationTransportHandler] [follow-cluster-0] received diff cluster state version [20] with uuid [JoQId75gSHO4hcVD3jJoWQ], diff size [200]	
» [2020-02-10T04:42:26,953][INFO ][o.e.x.c.a.AutoFollowCoordinator] [follow-cluster-0] starting auto-follower for remote cluster [leader_cluster]	
» [2020-02-10T04:42:26,973][DEBUG][o.e.c.c.C.CoordinatorPublication] [follow-cluster-0] publication ended successfully: Publication{term=1, version=20}	
» [2020-02-10T04:42:28,187][INFO ][o.e.c.m.MetaDataCreateIndexService] [follow-cluster-0] [.watches] creating index, cause [auto(bulk api)], templates [.watches], shards [1]/[0], mappings [_doc]	
» [2020-02-10T04:42:28,823][DEBUG][o.e.c.c.PublicationTransportHandler] [follow-cluster-0] received diff cluster state version [21] with uuid [_oP4O82MQmSgwB_ZAAbWAQ], diff size [632]	
» [2020-02-10T04:42:29,992][DEBUG][o.e.c.c.C.CoordinatorPublication] [follow-cluster-0] publication ended successfully: Publication{term=1, version=21}	
» [2020-02-10T04:42:30,515][DEBUG][o.e.c.c.PublicationTransportHandler] [follow-cluster-0] received diff cluster state version [22] with uuid [ijB_cpwpROufLOT8zrAalA], diff size [696]	
» [2020-02-10T04:42:30,736][DEBUG][o.e.c.c.C.CoordinatorPublication] [follow-cluster-0] publication ended successfully: Publication{term=1, version=22}	
» [2020-02-10T04:42:30,742][INFO ][o.e.x.c.a.AutoFollowCoordinator] [follow-cluster-0] Auto followed leader index [logs-20190101] as follow index [logs-20190101]	
» [2020-02-10T04:42:30,971][DEBUG][o.e.c.c.PublicationTransportHandler] [follow-cluster-0] received diff cluster state version [23] with uuid [YmVPGDA7S2GqFmT2uYHsKw], diff size [488]	
» [2020-02-10T04:42:31,279][DEBUG][o.e.c.c.C.CoordinatorPublication] [follow-cluster-0] publication ended successfully: Publication{term=1, version=23}	
» [2020-02-10T04:42:31,359][INFO ][o.e.c.m.MetaDataCreateIndexService] [follow-cluster-0] [.monitoring-es-7-2020.02.10] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[0], mappings [_doc]	
» [2020-02-10T04:42:31,399][DEBUG][o.e.c.c.PublicationTransportHandler] [follow-cluster-0] received diff cluster state version [24] with uuid [fFBDGH75TsaxWceTFiVmRw], diff size [2364]	
» [2020-02-10T04:42:31,625][DEBUG][o.e.c.c.C.CoordinatorPublication] [follow-cluster-0] publication ended successfully: Publication{term=1, version=24}	
» [2020-02-10T04:42:31,685][DEBUG][o.e.c.c.PublicationTransportHandler] [follow-cluster-0] received diff cluster state version [25] with uuid [AxrgdF1ZQ5KPo95aS-iIUg], diff size [376]	
» [2020-02-10T04:42:31,891][INFO ][o.e.x.w.WatcherService   ] [follow-cluster-0] reloading watcher, reason [new local watcher shard allocation ids], cancelled [0] queued tasks	
» [2020-02-10T04:42:31,971][DEBUG][o.e.c.c.C.CoordinatorPublication] [follow-cluster-0] publication ended successfully: Publication{term=1, version=25}	
» [2020-02-10T04:42:32,020][DEBUG][o.e.c.c.PublicationTransportHandler] [follow-cluster-0] received diff cluster state version [26] with uuid [_NZwucieTHSjqP0CctuN9Q], diff size [401]	
» [2020-02-10T04:42:32,425][DEBUG][o.e.c.c.C.CoordinatorPublication] [follow-cluster-0] publication ended successfully: Publication{term=1, version=26}	
» [2020-02-10T04:42:32,529][INFO ][o.e.c.m.MetaDataMappingService] [follow-cluster-0] [.watches/k-Z4J6-7TLiKpRE5mQ-zUQ] update_mapping [_doc]	
» [2020-02-10T04:42:32,617][INFO ][o.e.c.m.MetaDataMappingService] [follow-cluster-0] [.watches/k-Z4J6-7TLiKpRE5mQ-zUQ] update_mapping [_doc]	
» [2020-02-10T04:42:32,651][DEBUG][o.e.c.c.PublicationTransportHandler] [follow-cluster-0] received diff cluster state version [27] with uuid [b-1U9ouGSzeNm1drGn6o5g], diff size [711]	
» [2020-02-10T04:42:33,120][DEBUG][o.e.c.c.C.CoordinatorPublication] [follow-cluster-0] publication ended successfully: Publication{term=1, version=27}	
» [2020-02-10T04:42:33,583][DEBUG][o.e.c.c.PublicationTransportHandler] [follow-cluster-0] received diff cluster state version [28] with uuid [9kK0ncWrQ8mdiYpKhpev5g], diff size [223]	
» [2020-02-10T04:42:33,942][DEBUG][o.e.c.c.C.CoordinatorPublication] [follow-cluster-0] publication ended successfully: Publication{term=1, version=28}	
» [2020-02-10T04:42:33,956][DEBUG][o.e.c.c.PublicationTransportHandler] [follow-cluster-0] received diff cluster state version [29] with uuid [IHR6uMltREOqCnyBK5lkcA], diff size [458]	
» [2020-02-10T04:42:34,081][DEBUG][o.e.c.c.C.CoordinatorPublication] [follow-cluster-0] publication ended successfully: Publication{term=1, version=29}	
» [2020-02-10T04:42:34,100][DEBUG][o.e.c.c.PublicationTransportHandler] [follow-cluster-0] received diff cluster state version [30] with uuid [xIFLDwQRQ3e2TAWQFwRTDw], diff size [156]	
» [2020-02-10T04:42:34,225][DEBUG][o.e.c.c.C.CoordinatorPublication] [follow-cluster-0] publication ended successfully: Publication{term=1, version=30}	
» [2020-02-10T04:42:34.293944Z] [BUILD] Stopping node
@tvernum tvernum added >test-failure Triaged test failures from CI :Distributed Indexing/CCR Issues around the Cross Cluster State Replication features labels Feb 10, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/CCR)

@dnhatn dnhatn self-assigned this Feb 12, 2020
@ywelsch
Copy link
Contributor

ywelsch commented Feb 17, 2020

@dnhatn can this be closed now that #52284 is merged?

@dnhatn dnhatn mentioned this issue Feb 17, 2020
Merged
dnhatn added a commit that referenced this issue Feb 17, 2020
@dnhatn
Wait for no initializing shards in CCR tests (#52327)
c835c9d 
This commit restores the wait_for_no_initializing_shards check that was 
removed in #52284. We need this condition when waiting for the yellow
status on the follower because we bootstrap the primary shards of the
follower using the snapshot/restore.

Relates #52284
Closes #52126
dnhatn added a commit to dnhatn/elasticsearch that referenced this issue Feb 17, 2020
@dnhatn
Wait for no initializing shards in CCR tests (elastic#52327)
bcd9b39 
This commit restores the wait_for_no_initializing_shards check that was
removed in elastic#52284. We need this condition when waiting for the yellow
status on the follower because we bootstrap the primary shards of the
follower using the snapshot/restore.

Relates elastic#52284
Closes elastic#52126
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
:Distributed Indexing/CCR Issues around the Cross Cluster State Replication features >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Wait for no initializing shards in CCR tests dnhatn/elasticsearch
4 participants
@tvernum @ywelsch @dnhatn @elasticmachine