Add pragma to grace join runtime node with spilling #6253

lll-phill-lll · 2024-07-03T12:49:31Z

Changelog entry

Added pragma that enables spilling in grace join.
...

Changelog category

New feature

Additional information

...

github-actions · 2024-07-03T17:58:29Z

⚪ 2024-07-03 17:58:29 UTC Pre-commit check for 7a3076c has started.
⚪ 2024-07-03 18:01:48 UTC Build linux-x86_64-relwithdebinfo is running...
⚫ 2024-07-03 19:04:00 UTC Check cancelled

github-actions · 2024-07-03T18:05:58Z

⚪ 2024-07-03 18:05:58 UTC Pre-commit check for 7a3076c has started.
⚪ 2024-07-03 18:09:05 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-07-03 18:19:52 UTC Build successful.

github-actions · 2024-07-03T19:07:59Z

⚪ 2024-07-03 19:07:59 UTC Pre-commit check for c74b1ff has started.
⚪ 2024-07-03 19:10:34 UTC Build linux-x86_64-release-clang14 is running...
⚫ 2024-07-03 19:11:45 UTC Check cancelled

github-actions · 2024-07-03T19:08:20Z

⚪ 2024-07-03 19:08:20 UTC Pre-commit check for c74b1ff has started.
⚪ 2024-07-03 19:11:01 UTC Build linux-x86_64-release-asan is running...
⚫ 2024-07-03 19:11:31 UTC Check cancelled

github-actions · 2024-07-03T19:27:02Z

⚪ 2024-07-03 19:27:01 UTC Pre-commit check for 24d3e91 has started.
⚪ 2024-07-03 19:30:11 UTC Build linux-x86_64-release-asan is running...
🟢 2024-07-03 19:57:27 UTC Build successful.
⚪ 2024-07-03 19:57:38 UTC Tests are running...
🔴 2024-07-03 22:01:53 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15085	14475	0	92	340	178

🟢 2024-07-03 22:02:42 UTC ydbd size 5.3 GiB changed* by -97.9 MiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `425858c`	merge: `24d3e91`	diff	diff %
ydbd size	5 814 908 176 Bytes	5 712 275 584 Bytes	-97.9 MiB	-1.765%
ydbd stripped size	1 247 931 016 Bytes	1 235 334 728 Bytes	-12.0 MiB	-1.009%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-07-03T19:30:32Z

⚪ 2024-07-03 19:30:32 UTC Pre-commit check for 24d3e91 has started.
⚪ 2024-07-03 19:33:27 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-07-03 19:41:52 UTC Build successful.

github-actions · 2024-07-03T19:34:55Z

⚪ 2024-07-03 19:34:55 UTC Pre-commit check for 24d3e91 has started.
⚪ 2024-07-03 19:37:31 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-07-03 20:19:38 UTC Build successful.
⚪ 2024-07-03 20:19:51 UTC Tests are running...
🔴 2024-07-03 22:43:11 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
76115	62676	0	2	13426	11

🟢 2024-07-03 22:43:55 UTC ydbd size 8.2 GiB changed* by -144.8 MiB, which is <= 0 Bytes vs main: OK

ydbd size dash	main: `895e035`	merge: `24d3e91`	diff	diff %
ydbd size	8 927 071 568 Bytes	8 775 286 776 Bytes	-144.8 MiB	-1.700%
ydbd stripped size	486 819 816 Bytes	482 686 984 Bytes	-3.9 MiB	-0.849%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

ydb/library/yql/minikql/comp_nodes/mkql_grace_join.cpp

ydb/library/yql/providers/dq/common/yql_dq_settings.cpp

ydb/library/yql/dq/tasks/dq_task_program.cpp

github-actions · 2024-07-04T12:05:41Z

⚪ 2024-07-04 12:05:40 UTC Pre-commit check for 2bcae57 has started.
⚪ 2024-07-04 12:08:36 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-07-04 12:37:00 UTC Build successful.

github-actions · 2024-07-04T12:05:57Z

⚪ 2024-07-04 12:05:57 UTC Pre-commit check for 2bcae57 has started.
⚪ 2024-07-04 12:08:46 UTC Build linux-x86_64-release-asan is running...
🟢 2024-07-04 12:45:42 UTC Build successful.
⚪ 2024-07-04 12:45:58 UTC Tests are running...
🔴 2024-07-04 13:09:04 UTC Test run completed, no test results found for commit 621daf0. Please check test log.
🟡 2024-07-04 13:09:57 UTC ydbd size 5.3 GiB changed* by +365.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `bcc7d5e`	merge: `2bcae57`	diff	diff %
ydbd size	5 712 433 376 Bytes	5 712 807 400 Bytes	+365.3 KiB	+0.007%
ydbd stripped size	1 235 350 184 Bytes	1 235 428 264 Bytes	+76.2 KiB	+0.006%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}
⚫ 2024-07-04 13:09:58 UTC Check cancelled

ydb/library/yql/minikql/mkql_node.h

ydb/library/yql/providers/dq/common/yql_dq_settings.h

ydb/library/yql/providers/dq/provider/exec/yql_dq_exectransformer.cpp

github-actions · 2024-07-04T17:05:25Z

⚪ 2024-07-04 17:05:24 UTC Pre-commit check for da69c53 has started.
⚪ 2024-07-04 17:08:07 UTC Build linux-x86_64-release-asan is running...
🟢 2024-07-04 17:39:41 UTC Build successful.
⚪ 2024-07-04 17:39:56 UTC Tests are running...
🔴 2024-07-04 19:52:33 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15167	14566	0	91	340	170

🟢 2024-07-04 19:53:15 UTC ydbd size 5.3 GiB changed* by +91.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `29e1e45`	merge: `da69c53`	diff	diff %
ydbd size	5 712 635 128 Bytes	5 712 728 912 Bytes	+91.6 KiB	+0.002%
ydbd stripped size	1 235 400 808 Bytes	1 235 415 176 Bytes	+14.0 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-07-04T17:07:01Z

⚪ 2024-07-04 17:07:01 UTC Pre-commit check for da69c53 has started.
⚪ 2024-07-04 17:09:53 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-07-04 17:30:28 UTC Build successful.

github-actions · 2024-07-04T17:31:55Z

⚪ 2024-07-04 17:31:55 UTC Pre-commit check for da69c53 has started.
⚪ 2024-07-04 17:34:58 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-07-04 18:22:16 UTC Build successful.
⚪ 2024-07-04 18:22:41 UTC Tests are running...
🔴 2024-07-04 20:36:19 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
76208	62763	0	2	13429	14

🟡 2024-07-04 20:36:59 UTC ydbd size 8.2 GiB changed* by +316.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `4eb9bf7`	merge: `da69c53`	diff	diff %
ydbd size	8 775 756 672 Bytes	8 776 081 184 Bytes	+316.9 KiB	+0.004%
ydbd stripped size	482 728 040 Bytes	482 731 560 Bytes	+3.4 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

rvu1024 · 2024-07-05T08:40:17Z

ydb/library/yql/providers/dq/provider/exec/yql_dq_exectransformer.cpp

+            NDq::TSpillingSettings spillingSettings{State->Settings->GetEnabledSpillingNodes()};
            *lambda = NDq::BuildProgram(
                programLambda, *paramsType, compiler, typeEnv, *State->FunctionRegistry,
-                ctx, fakeReads);
+                ctx, fakeReads, spillingSettings);


Этот код зовется из HandleResult, который в том числе может исполнять лямбду локально. Тут нужно либо отключать сплиинг для локального исполнения, либо убедиться, что все отработает номрально без spilling service под ногами

Спиллинг не должен включаться без spilling service под ногами. Нужно добавить проверку в TSpillingTransformProvider, чтобы спиллинг ноды не включались, если спиллинга нет.

Предлагаю это сделать в PR по добавлению прагмы в widecombine, чтобы не ждать 3 часа тестов

lll-phill-lll added 2 commits July 3, 2024 13:41

init

da58d3b

added pseudocode of solution

629070c

github-actions bot added the new-feature label Jul 3, 2024

This comment was marked as resolved.

Sign in to view

This comment was marked as outdated.

Sign in to view

add duntime nodes

9c8982c

lll-phill-lll marked this pull request as ready for review July 3, 2024 16:03

lll-phill-lll requested review from a team as code owners July 3, 2024 16:03

lll-phill-lll added 2 commits July 3, 2024 19:20

fixup

9c09778

renaming works

870b68e

This comment was marked as outdated.

Sign in to view

add TSpillingSettings

69259bd

fixup

6111922

lll-phill-lll requested review from rvu1024 and Darych July 3, 2024 19:13

Darych previously approved these changes Jul 4, 2024

View reviewed changes

rvu1024 reviewed Jul 4, 2024

View reviewed changes

ydb/library/yql/minikql/comp_nodes/mkql_grace_join.cpp Outdated Show resolved Hide resolved

ydb/library/yql/providers/dq/common/yql_dq_settings.cpp Outdated Show resolved Hide resolved

ydb/library/yql/dq/tasks/dq_task_program.cpp Outdated Show resolved Hide resolved

review fixes

621daf0

lll-phill-lll dismissed Darych’s stale review via 621daf0 July 4, 2024 12:03

rvu1024 reviewed Jul 4, 2024

View reviewed changes

ydb/library/yql/minikql/mkql_node.h Outdated Show resolved Hide resolved

ydb/library/yql/providers/dq/common/yql_dq_settings.h Outdated Show resolved Hide resolved

ydb/library/yql/providers/dq/provider/exec/yql_dq_exectransformer.cpp Outdated Show resolved Hide resolved

review fixes

b65e777

aakulaga-ydb requested review from rvu1024 and Darych July 5, 2024 08:28

rvu1024 reviewed Jul 5, 2024

View reviewed changes

rvu1024 approved these changes Jul 5, 2024

View reviewed changes

lll-phill-lll merged commit c9e20ce into ydb-platform:main Jul 5, 2024
8 of 12 checks passed

StekPerepolnen mentioned this pull request Jul 5, 2024

hc remove time difference public api mention #6335

Merged

This was referenced Jul 8, 2024

YQ-3367 frozen fq yt-kqp test list #6397

Merged

Kqprun added request time info printing #6479

Merged

YQ-3405 fixed endless retries for external error #6626

Merged

YQ-3345 fixed WM counters and unit test #6643

Merged

GrigoriyPA mentioned this pull request Jul 24, 2024

YQ WM fix for sls feature flag #7057

Merged

lll-phill-lll added a commit to lll-phill-lll/ydb that referenced this pull request Sep 6, 2024

Add pragma to grace join runtime node with spilling (ydb-platform#6253)

ec8be9e

lll-phill-lll added a commit to lll-phill-lll/ydb that referenced this pull request Sep 6, 2024

Add pragma to grace join runtime node with spilling (ydb-platform#6253)

a7ffab9

lll-phill-lll added a commit to lll-phill-lll/ydb that referenced this pull request Sep 10, 2024

Add pragma to grace join runtime node with spilling (ydb-platform#6253)

aaf99bc

lll-phill-lll added a commit to lll-phill-lll/ydb that referenced this pull request Sep 11, 2024

Add pragma to grace join runtime node with spilling (ydb-platform#6253)

dd17c82

lll-phill-lll added a commit that referenced this pull request Sep 12, 2024

Add pragma to grace join runtime node with spilling (#6253)

097def5

lll-phill-lll added a commit to lll-phill-lll/ydb that referenced this pull request Sep 13, 2024

Add pragma to grace join runtime node with spilling (ydb-platform#6253)

846bb02

lll-phill-lll added a commit to lll-phill-lll/ydb that referenced this pull request Sep 15, 2024

Add pragma to grace join runtime node with spilling (ydb-platform#6253)

8f23bf4

lll-phill-lll added a commit to lll-phill-lll/ydb that referenced this pull request Sep 15, 2024

Add pragma to grace join runtime node with spilling (ydb-platform#6253)

3057f54

lll-phill-lll mentioned this pull request Sep 18, 2024

Stable 24 3 8 analytics #9022

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pragma to grace join runtime node with spilling #6253

Add pragma to grace join runtime node with spilling #6253

lll-phill-lll commented Jul 3, 2024 •

edited

Loading

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Jul 3, 2024 •

edited

Loading

github-actions bot commented Jul 3, 2024 •

edited

Loading

github-actions bot commented Jul 3, 2024 •

edited

Loading

github-actions bot commented Jul 3, 2024 •

edited

Loading

github-actions bot commented Jul 3, 2024 •

edited

Loading

github-actions bot commented Jul 3, 2024 •

edited

Loading

github-actions bot commented Jul 3, 2024 •

edited

Loading

github-actions bot commented Jul 4, 2024 •

edited

Loading

github-actions bot commented Jul 4, 2024 •

edited

Loading

github-actions bot commented Jul 4, 2024 •

edited

Loading

github-actions bot commented Jul 4, 2024 •

edited

Loading

github-actions bot commented Jul 4, 2024 •

edited

Loading

rvu1024 Jul 5, 2024

lll-phill-lll Jul 5, 2024

Add pragma to grace join runtime node with spilling #6253

Add pragma to grace join runtime node with spilling #6253

Conversation

lll-phill-lll commented Jul 3, 2024 • edited Loading

Changelog entry

Changelog category

Additional information

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Jul 3, 2024 • edited Loading

github-actions bot commented Jul 3, 2024 • edited Loading

github-actions bot commented Jul 3, 2024 • edited Loading

github-actions bot commented Jul 3, 2024 • edited Loading

github-actions bot commented Jul 3, 2024 • edited Loading

github-actions bot commented Jul 3, 2024 • edited Loading

github-actions bot commented Jul 3, 2024 • edited Loading

github-actions bot commented Jul 4, 2024 • edited Loading

github-actions bot commented Jul 4, 2024 • edited Loading

github-actions bot commented Jul 4, 2024 • edited Loading

github-actions bot commented Jul 4, 2024 • edited Loading

github-actions bot commented Jul 4, 2024 • edited Loading

rvu1024 Jul 5, 2024

Choose a reason for hiding this comment

lll-phill-lll Jul 5, 2024

Choose a reason for hiding this comment

lll-phill-lll commented Jul 3, 2024 •

edited

Loading

github-actions bot commented Jul 3, 2024 •

edited

Loading

github-actions bot commented Jul 3, 2024 •

edited

Loading

github-actions bot commented Jul 3, 2024 •

edited

Loading

github-actions bot commented Jul 3, 2024 •

edited

Loading

github-actions bot commented Jul 3, 2024 •

edited

Loading

github-actions bot commented Jul 3, 2024 •

edited

Loading

github-actions bot commented Jul 3, 2024 •

edited

Loading

github-actions bot commented Jul 4, 2024 •

edited

Loading

github-actions bot commented Jul 4, 2024 •

edited

Loading

github-actions bot commented Jul 4, 2024 •

edited

Loading

github-actions bot commented Jul 4, 2024 •

edited

Loading

github-actions bot commented Jul 4, 2024 •

edited

Loading