Skip to content

Commit 712dfc2

Browse files
lievanlievan
and
lievan
authored
chore(llmobs): fix flaky evaluator runner log test (#13118)
Increase the interval for evaluator runner being used in the buffer limit test to try to make it less flaky previously it was using `active_evaluator_runner` fixture which had a interval of 0.01 ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: lievan <[email protected]>
1 parent 3862cef commit 712dfc2

File tree

2 files changed

+6
-4
lines changed

2 files changed

+6
-4
lines changed

tests/llmobs/conftest.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,7 @@ def mock_writer_logs():
9797
def mock_evaluator_logs():
9898
with mock.patch("ddtrace.llmobs._evaluators.runner.logger") as m:
9999
yield m
100+
m.reset_mock()
100101

101102

102103
@pytest.fixture

tests/llmobs/test_llmobs_evaluator_runner.py

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,6 @@
1111
from ddtrace.trace import Span
1212
from tests.llmobs._utils import DummyEvaluator
1313
from tests.llmobs._utils import _dummy_evaluator_eval_metric_event
14-
from tests.utils import flaky
1514
from tests.utils import override_env
1615
from tests.utils import override_global_config
1716

@@ -31,10 +30,12 @@ def test_evaluator_runner_start(mock_evaluator_logs, active_evaluator_runner):
3130
mock_evaluator_logs.debug.assert_has_calls([mock.call("started %r", "EvaluatorRunner")])
3231

3332

34-
@flaky(1744053478)
35-
def test_evaluator_runner_buffer_limit(mock_evaluator_logs, active_evaluator_runner):
33+
def test_evaluator_runner_buffer_limit(mock_evaluator_logs):
34+
evaluator_runner = EvaluatorRunner(interval=1, llmobs_service=mock.MagicMock())
35+
evaluator_runner.evaluators.append(DummyEvaluator(llmobs_service=mock.MagicMock()))
36+
evaluator_runner.start()
3637
for _ in range(1001):
37-
active_evaluator_runner.enqueue({}, DUMMY_SPAN)
38+
evaluator_runner.enqueue({}, DUMMY_SPAN)
3839
mock_evaluator_logs.warning.assert_called_with(
3940
"%r event buffer full (limit is %d), dropping event", "EvaluatorRunner", 1000
4041
)

0 commit comments

Comments
 (0)