[Service Bus & Event Hubs] Improve livetest test stability

[yunhaoling]

• finding if there's any tests repeatedly failing in our livetest
• whether it's an issue in SDK or VM agents' problem
• if not issue in SDK, improve the test

[swathipil]

check devops and check if something can be combined with stress test onboarding

[swathipil]

SB test flaky: test_queue_send_twice: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1321365&view=logs&j=217ad987-cc73-5dcd-abbd-eb23972de726&t=f10366f4-42a0-5bf3-0a2a-6be221a21989&l=1873

[swathipil]

update:

• have been checking the nightly test runs, and there are no consistently flaky tests other than one above. Just some other ones only failing once from what I've seen:

EH:
tests/livetest/synctests/test_consumer_client.py::test_receive_batch_no_max_wait_time

SB:
FAILED tests/test_subscriptions.py::ServiceBusSubscriptionTests::test_subscription_by_servicebus_client_receive_batch_with_deadletter: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1326700&view=logs&j=fa284728-64c9-5de0-4442-32743d14c85e&t=b93d123d-eb8b-5272-3a5e-e2a5347ca4ca&l=2272
FAILED tests/async_tests/test_queues_async.py::ServiceBusQueueAsyncTests::test_async_queue_by_servicebus_client_browse_messages_with_receiver
FAILED tests\test_queues.py::ServiceBusQueueTests::test_queue_by_queue_client_conn_str_receive_handler_with_autolockrenew - https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1326700&view=logs&j=217ad987-cc73-5dcd-abbd-eb23972de726&t=f10366f4-42a0-5bf3-0a2a-6be221a21989&l=2126
FAILED tests/test_queues.py::ServiceBusQueueTests::test_queue_send_dict_messages_scheduled
FAILED tests/test_queues.py::ServiceBusQueueTests::test_queue_operation_negative

[swathipil]

check 12/28:
FAILED tests/async_tests/test_queues_async.py::ServiceBusQueueAsyncTests::test_async_queue_by_queue_client_conn_str_receive_handler_peeklock

12/31:
FAILED tests/test_sessions.py::ServiceBusSessionTests::test_session_by_servicebus_client_session_pool
FAILED tests/test_sessions.py::ServiceBusSessionTests::test_session_by_servicebus_client_session_pool
FAILED tests/test_sessions.py::ServiceBusSessionTests::test_session_connection_failure_is_idempotent
FAILED tests/async_tests/test_sessions_async.py::ServiceBusAsyncSessionTests::test_async_session_connection_failure_is_idempotent
FAILED tests/async_tests/test_queues_async.py::ServiceBusQueueAsyncTests::test_async_queue_receiver_respects_max_wait_time_overrides
FAILED tests/async_tests/test_sessions_async.py::ServiceBusAsyncSessionTests::test_async_session_by_servicebus_client_session_pool
FAILED tests/async_tests/test_sessions_async.py::ServiceBusAsyncSessionTests::test_async_session_by_session_client_conn_str_receive_handler_with_no_session
FAILED tests/async_tests/test_sessions_async.py::ServiceBusAsyncSessionTests::test_async_session_cancel_scheduled_messages
FAILED tests/async_tests/test_sessions_async.py::ServiceBusAsyncSessionTests::test_async_session_cancel_scheduled_messages
FAILED tests/test_sessions.py::ServiceBusSessionTests::test_session_by_servicebus_client_session_pool
FAILED tests/async_tests/test_queues_async.py::ServiceBusQueueAsyncTests::test_async_queue_receiver_respects_max_wait_time_overrides
FAILED tests/async_tests/test_sessions_async.py::ServiceBusAsyncSessionTests::test_async_session_by_servicebus_client_session_pool
FAILED tests/async_tests/test_sessions_async.py::ServiceBusAsyncSessionTests::test_async_session_by_session_client_conn_str_receive_handler_with_no_session
FAILED tests/async_tests/test_subscriptions_async.py::ServiceBusSubscriptionAsyncTests::test_topic_by_servicebus_client_receive_batch_with_deadletter
FAILED tests/test_queues.py::ServiceBusQueueTests::test_queue_receive_keep_conn_alive
FAILED tests/test_sessions.py::ServiceBusSessionTests::test_session_by_servicebus_client_session_pool
FAILED tests/test_sessions.py::ServiceBusSessionTests::test_session_connection_failure_is_idempotent
FAILED tests/async_tests/test_queues_async.py::ServiceBusQueueAsyncTests::test_queue_receive_keep_conn_alive_async
FAILED tests/mgmt_tests/test_mgmt_namespaces.py::ServiceBusManagementClientNamespaceTests::test_mgmt_namespace_get_properties

FIX: async/sync test_session_by_servicebus_client_session_pool: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1336668&view=logs&j=a5c73adf-21c4-51b0-3477-575974909b75&t=920caeb1-8790-523a-08a8-f5f4e4c5560b&l=1787

[swathipil]

A lot of these are failing b/c of this error:
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1343200&view=logs&j=fe7804ca-eaac-501a-67e8-e2604a50cb52&t=581043c6-4578-52e9-e224-2083bb8809be&l=3251
and this: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1343200&view=logs&j=fe7804ca-eaac-501a-67e8-e2604a50cb52&t=581043c6-4578-52e9-e224-2083bb8809be&l=2979
And this one with Auth Timeout:
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1343200&view=logs&j=fe7804ca-eaac-501a-67e8-e2604a50cb52&t=581043c6-4578-52e9-e224-2083bb8809be&l=1947

[swathipil]

receive_handler_with_auto_lock_renew:
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1349066&view=logs&j=36c1d9f8-0c10-5405-9a7f-ac15f4c32de9&t=654dbbb3-fbc2-5bfe-7f7d-126a91b09b22&l=2191

[swathipil]

iter_with_defer_not_found:
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1355297&view=logs&j=fa284728-64c9-5de0-4442-32743d14c85e&t=b93d123d-eb8b-5272-3a5e-e2a5347ca4ca&l=2009
test_message_lock_renew:
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1358691&view=logs&j=217ad987-cc73-5dcd-abbd-eb23972de726&t=f10366f4-42a0-5bf3-0a2a-6be221a21989&l=4644

[swathipil]

The error in test_session_by_servicebus_client_session_pool with Cannot open log for source 'Microsoft.ServiceBus' seems to be an error with the service. Similar issue was filed here : Azure/azure-sdk-for-net#27067

For now, ignoring this error in the test.

EDIT: It looks like this error is happening in every test where .get_queue_receiver(...session_id=NEXT_AVAILABLE_SESSION) and seems to be after this call results in an OperationTimeoutError. For ex:

azure-sdk-for-python/sdk/servicebus/azure-servicebus/tests/test_sessions.py

Line 259 in a1deaf3

with sb_client.get_queue_receiver(servicebus_queue.name,

[swathipil]

nightly runs have been green for a week so closing

[swathipil]

more failures 3/7: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1413650&view=results

[yunhaoling]

success criteria: 90% pass rate for the past two weeks

[swathipil]

most flaky EH tests:

• sync and async test_buffered_producer.py:
  • test_basic_send_single_events_round_robin[--False]
  • test_long_wait_small_buffer:

        with producer:
            for i in range(100):
                producer.send_event(EventData("test"))
    
        time.sleep(60)
    
        assert not on_error.err
        assert sum([len(sent_events[key]) for key in sent_events]) == 100
>       assert sum([len(received_events[key]) for key in received_events]) == 100
E       assert 50 == 100
E         +50
E         -100

• test_basic_send_batch_events_round_robin[--False]

            if not flush_after_sending and not close_after_sending:
                # ensure it's buffered sending
                for pid in partitions:
                    assert len(sent_events[pid]) < each_partition_cnt
                assert sum([len(sent_events[pid]) for pid in partitions]) < total_events_cnt
                # give some time for producer to complete sending and consumer to complete receiving
            else:
                if flush_after_sending:
                    producer.flush()
                if close_after_sending:
                    producer.close()
                # ensure all events are sent
                assert sum([len(sent_events[pid]) for pid in partitions]) == total_events_cnt
    
            time.sleep(10)
>           assert len(sent_events) == len(received_events) == partitions_cnt
E           assert 2 == 1
E             +2
E             -1

• sync and async test_send.py:
  • test_send_with_partition_key[***]:

source = "amqps://{}/{}/ConsumerGroups/{}/Partitions/{}".format(
                        live_eventhub['hostname'],
                        live_eventhub['event_hub'],
                        live_eventhub['consumer_group'],
                        index)
                    partition = uamqp.ReceiveClient(source, auth=sas_auth, debug=***, timeout=0, prefetch=500)
                    reconnect_receivers.append(partition)
                    retry_total += 1
            if retry_total == 3:
                raise OperationTimeoutError(f"Exhausted retries for receiving from {live_eventhub['hostname']}.")
    
        for r in reconnect_receivers:
            r.close()
    
>       assert single_cnt == 60
E       assert 30 == 60
E         +30
E         -60

In general, only receiving half of expected events when using a ConsumerClient to receive, instead of receivers in the conftest fixture.