Tests for wide combiner with spilling #6880

lll-phill-lll · 2024-07-19T14:49:55Z

Changelog entry

Added tests for wide combiner with spilling
...

Changelog category

Not for changelog (changelog entry is not required)

Additional information

...

github-actions · 2024-07-19T14:51:11Z

⚪ 2024-07-19 14:51:11 UTC Pre-commit check for 64e945c has started.
⚪ 2024-07-19 14:53:55 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🔴 2024-07-19 15:42:53 UTC Test run completed, no test results found for commit 170b5d9.
⚫ 2024-07-19 15:43:13 UTC Check cancelled

github-actions · 2024-07-19T14:51:11Z

⚪ 2024-07-19 14:51:11 UTC Pre-commit check for 64e945c has started.
⚪ 2024-07-19 14:53:56 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-19 14:59:25 UTC Build successful.

github-actions · 2024-07-19T14:51:11Z

⚪ 2024-07-19 14:51:11 UTC Pre-commit check for 64e945c has started.
⚪ 2024-07-19 14:53:59 UTC Build+Tests linux-x86_64-release-asan is running...
🔴 2024-07-19 15:42:54 UTC Test run completed, no test results found for commit 170b5d9.
⚫ 2024-07-19 15:43:16 UTC Check cancelled

lll-phill-lll · 2024-07-19T15:46:24Z

ydb/library/yql/minikql/comp_nodes/mkql_wide_combine.cpp

-        return false;
-        // TODO: YQL-18033
-        // return !HasMemoryForProcessing();
+        return !HasMemoryForProcessing();


Please note that the hardcoded value is removed here

github-actions · 2024-07-19T15:46:41Z

⚪ 2024-07-19 15:46:41 UTC Pre-commit check for 451d38c has started.
⚫ 2024-07-19 15:49:30 UTC Check cancelled

github-actions · 2024-07-19T15:46:47Z

⚪ 2024-07-19 15:46:46 UTC Pre-commit check for 451d38c has started.
⚫ 2024-07-19 15:49:31 UTC Check cancelled

github-actions · 2024-07-19T15:46:53Z

⚪ 2024-07-19 15:46:52 UTC Pre-commit check for 451d38c has started.
⚫ 2024-07-19 15:49:33 UTC Check cancelled

github-actions · 2024-07-19T15:49:18Z

🔴 2024-07-19 15:49:17 UTC Test run completed, no test results found for commit 8167088.

github-actions · 2024-07-19T15:50:56Z

⚪ 2024-07-19 15:50:56 UTC Pre-commit check for 3cf83e4 has started.
⚪ 2024-07-19 15:53:43 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-19 15:59:27 UTC Build successful.

github-actions · 2024-07-19T15:50:57Z

⚪ 2024-07-19 15:50:57 UTC Pre-commit check for 3cf83e4 has started.
⚪ 2024-07-19 15:53:42 UTC Build+Tests linux-x86_64-release-asan is running...
🟢 2024-07-19 17:58:28 UTC Build successful.
🔴 2024-07-19 18:01:01 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11983	11375	0	97	335	176

🔴 2024-07-19 18:01:50 UTC ydbd size 5.2 GiB changed* by +8.5 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash	main: `19153a4`	merge: `3cf83e4`	diff	diff %
ydbd size	5 546 451 224 Bytes	5 555 350 760 Bytes	+8.5 MiB	+0.160%
ydbd stripped size	1 195 225 712 Bytes	1 196 909 136 Bytes	+1.6 MiB	+0.141%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-07-19T15:51:40Z

⚪ 2024-07-19 15:51:40 UTC Pre-commit check for 3cf83e4 has started.
⚪ 2024-07-19 15:54:31 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🟢 2024-07-19 18:07:16 UTC Build successful.
🔴 2024-07-19 18:15:38 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
73065	59609	0	4	13439	13

🔴 2024-07-19 18:16:21 UTC ydbd size 8.1 GiB changed* by +13.1 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash	main: `19153a4`	merge: `3cf83e4`	diff	diff %
ydbd size	8 641 993 624 Bytes	8 655 738 472 Bytes	+13.1 MiB	+0.159%
ydbd stripped size	470 492 384 Bytes	471 109 408 Bytes	+602.6 KiB	+0.131%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-07-21T14:39:26Z

⚪ 2024-07-21 14:39:26 UTC Pre-commit check for 1c6e126 has started.
⚪ 2024-07-21 14:42:11 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-21 14:48:02 UTC Build successful.

github-actions · 2024-07-21T14:39:37Z

⚪ 2024-07-21 14:39:36 UTC Pre-commit check for 1c6e126 has started.
⚪ 2024-07-21 14:42:22 UTC Build+Tests linux-x86_64-release-asan is running...
🟢 2024-07-21 16:48:02 UTC Build successful.
🔴 2024-07-21 16:50:13 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11982	11377	0	94	338	173

🟢 2024-07-21 16:51:01 UTC ydbd size 5.2 GiB changed* by +82.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `5e95651`	merge: `1c6e126`	diff	diff %
ydbd size	5 556 133 632 Bytes	5 556 218 344 Bytes	+82.7 KiB	+0.002%
ydbd stripped size	1 197 007 984 Bytes	1 197 029 424 Bytes	+20.9 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-07-21T14:42:20Z

⚪ 2024-07-21 14:42:19 UTC Pre-commit check for 1c6e126 has started.
⚪ 2024-07-21 14:45:16 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🟢 2024-07-21 17:00:10 UTC Build successful.
🔴 2024-07-21 17:08:22 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
73063	59604	0	5	13439	15

🟡 2024-07-21 17:09:05 UTC ydbd size 8.1 GiB changed* by +1.1 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `1c15550`	merge: `1c6e126`	diff	diff %
ydbd size	8 656 109 112 Bytes	8 657 276 824 Bytes	+1.1 MiB	+0.013%
ydbd stripped size	471 115 872 Bytes	471 149 856 Bytes	+33.2 KiB	+0.007%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-07-22T10:39:49Z

⚪ 2024-07-22 10:39:49 UTC Pre-commit check for c413db0 has started.
⚪ 2024-07-22 10:42:32 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-22 10:59:22 UTC Build successful.

github-actions · 2024-07-22T10:40:35Z

⚪ 2024-07-22 10:40:35 UTC Pre-commit check for c413db0 has started.
⚪ 2024-07-22 10:43:20 UTC Build+Tests linux-x86_64-release-asan is running...
🟢 2024-07-22 13:02:45 UTC Build successful.
🔴 2024-07-22 13:06:10 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11981	11373	0	93	341	174

🟢 2024-07-22 13:07:08 UTC ydbd size 5.2 GiB changed* by +86.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `0fd2795`	merge: `c413db0`	diff	diff %
ydbd size	5 556 859 008 Bytes	5 556 947 824 Bytes	+86.7 KiB	+0.002%
ydbd stripped size	1 196 988 848 Bytes	1 197 014 384 Bytes	+24.9 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-07-22T10:41:55Z

⚪ 2024-07-22 10:41:55 UTC Pre-commit check for c413db0 has started.
⚪ 2024-07-22 10:45:12 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🟢 2024-07-22 12:56:27 UTC Build successful.
🔴 2024-07-22 13:04:51 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
73071	59603	0	10	13438	20

🟡 2024-07-22 13:05:38 UTC ydbd size 8.1 GiB changed* by +108.1 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `0fd2795`	merge: `c413db0`	diff	diff %
ydbd size	8 658 392 912 Bytes	8 658 503 640 Bytes	+108.1 KiB	+0.001%
ydbd stripped size	471 146 656 Bytes	471 156 000 Bytes	+9.1 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

Darych · 2024-07-22T15:49:04Z

ydb/library/yql/minikql/comp_nodes/mkql_wide_combine.cpp

+        }
+        logmsg << (used/1_MB) << "MB/" << (limit/1_MB) << "MB";
+
+        YQL_LOG(INFO) << logmsg;


Maybe we want to print it on Debug level?

Discussed that this log would be written once per task, so we decided to leave the INFO level.

Darych · 2024-07-22T15:49:43Z

ydb/library/yql/minikql/comp_nodes/mkql_wide_combine.cpp

@@ -688,10 +696,12 @@ class TSpillingSupportState : public TComputationValue<TSpillingSupportState> {
    void SwitchMode(EOperatingMode mode) {
        switch(mode) {
            case EOperatingMode::InMemory: {
+                YQL_LOG(INFO) << "switching Memory mode to InMemory";


And here also in Debug?

Discussed that this log would be written once per task, so we decided to leave the INFO level.

lll-phill-lll added 6 commits July 19, 2024 17:46

Added tests for widecombiner with spilling

c330758

fixup

3413bac

fixupo

896b505

first test works

bde88f9

added more tests

1e82ca6

added all tests

170b5d9

github-actions bot added the not-for-changelog label Jul 19, 2024

lll-phill-lll added 3 commits July 19, 2024 17:54

removed redundand changes

6c1f1c9

disable test

a442036

removed zero division in log

8167088

lll-phill-lll marked this pull request as ready for review July 19, 2024 15:43

lll-phill-lll requested a review from a team as a code owner July 19, 2024 15:43

lll-phill-lll requested a review from Darych July 19, 2024 15:43

lll-phill-lll commented Jul 19, 2024

View reviewed changes

fixup

c7396a0

just better

f18e7fd

fixup

48b00f6

Darych approved these changes Jul 22, 2024

View reviewed changes

lll-phill-lll merged commit 6123ecb into ydb-platform:main Jul 22, 2024
8 of 12 checks passed

This was referenced Jul 23, 2024

YQ-3446 add queued time into query stats #6965

Merged

YQ-3459 fix resource pools permissions #6989

Merged

YQ WM fix for sls feature flag #7057

Merged

lll-phill-lll added a commit to lll-phill-lll/ydb that referenced this pull request Sep 9, 2024

Tests for wide combiner with spilling (ydb-platform#6880)

a5200c0

lll-phill-lll added a commit to lll-phill-lll/ydb that referenced this pull request Sep 10, 2024

Tests for wide combiner with spilling (ydb-platform#6880)

cc55fa6

lll-phill-lll added a commit to lll-phill-lll/ydb that referenced this pull request Sep 11, 2024

Tests for wide combiner with spilling (ydb-platform#6880)

28cdca7

lll-phill-lll added a commit that referenced this pull request Sep 12, 2024

Tests for wide combiner with spilling (#6880)

557e138

lll-phill-lll added a commit to lll-phill-lll/ydb that referenced this pull request Sep 13, 2024

Tests for wide combiner with spilling (ydb-platform#6880)

13b72f5

lll-phill-lll added a commit to lll-phill-lll/ydb that referenced this pull request Sep 13, 2024

Tests for wide combiner with spilling (ydb-platform#6880)

8348cfd

lll-phill-lll added a commit to lll-phill-lll/ydb that referenced this pull request Sep 15, 2024

Tests for wide combiner with spilling (ydb-platform#6880)

cb117bc

lll-phill-lll added a commit to lll-phill-lll/ydb that referenced this pull request Sep 15, 2024

Tests for wide combiner with spilling (ydb-platform#6880)

e410ef9

lll-phill-lll mentioned this pull request Sep 18, 2024

Stable 24 3 8 analytics #9022

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tests for wide combiner with spilling #6880

Tests for wide combiner with spilling #6880

lll-phill-lll commented Jul 19, 2024

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024 •

edited

Loading

lll-phill-lll Jul 19, 2024

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 21, 2024 •

edited

Loading

github-actions bot commented Jul 21, 2024 •

edited

Loading

github-actions bot commented Jul 21, 2024 •

edited

Loading

github-actions bot commented Jul 22, 2024 •

edited

Loading

github-actions bot commented Jul 22, 2024 •

edited

Loading

github-actions bot commented Jul 22, 2024 •

edited

Loading

Darych Jul 22, 2024

lll-phill-lll Jul 22, 2024

Darych Jul 22, 2024

lll-phill-lll Jul 22, 2024

Tests for wide combiner with spilling #6880

Tests for wide combiner with spilling #6880

Conversation

lll-phill-lll commented Jul 19, 2024

Changelog entry

Changelog category

Additional information

github-actions bot commented Jul 19, 2024 • edited Loading

github-actions bot commented Jul 19, 2024 • edited Loading

github-actions bot commented Jul 19, 2024 • edited Loading

lll-phill-lll Jul 19, 2024

Choose a reason for hiding this comment

github-actions bot commented Jul 19, 2024 • edited Loading

github-actions bot commented Jul 19, 2024 • edited Loading

github-actions bot commented Jul 19, 2024 • edited Loading

github-actions bot commented Jul 19, 2024

github-actions bot commented Jul 19, 2024 • edited Loading

github-actions bot commented Jul 19, 2024 • edited Loading

github-actions bot commented Jul 19, 2024 • edited Loading

github-actions bot commented Jul 21, 2024 • edited Loading

github-actions bot commented Jul 21, 2024 • edited Loading

github-actions bot commented Jul 21, 2024 • edited Loading

github-actions bot commented Jul 22, 2024 • edited Loading

github-actions bot commented Jul 22, 2024 • edited Loading

github-actions bot commented Jul 22, 2024 • edited Loading

Darych Jul 22, 2024

Choose a reason for hiding this comment

lll-phill-lll Jul 22, 2024

Choose a reason for hiding this comment

Darych Jul 22, 2024

Choose a reason for hiding this comment

lll-phill-lll Jul 22, 2024

Choose a reason for hiding this comment

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024 •

edited

Loading

github-actions bot commented Jul 21, 2024 •

edited

Loading

github-actions bot commented Jul 21, 2024 •

edited

Loading

github-actions bot commented Jul 21, 2024 •

edited

Loading

github-actions bot commented Jul 22, 2024 •

edited

Loading

github-actions bot commented Jul 22, 2024 •

edited

Loading

github-actions bot commented Jul 22, 2024 •

edited

Loading