Skip to content

Added another cost based optimization level #8348

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Aug 28, 2024
Merged

Added another cost based optimization level #8348

merged 4 commits into from
Aug 28, 2024

Conversation

pavelvelikhov
Copy link
Collaborator

@pavelvelikhov pavelvelikhov commented Aug 27, 2024 •
edited
Loading

Changelog entry

Новая логика уровней оптимизаций:

0 - CBO выключен
1 - CBO не включается, но считаются предсказания оптимизатора
2 - CBO включается в полном виде только для запросов, где есть OLAP таблицы
3 - CBO включается для всех запросов, но для даташардов предпочитает lookup join.
4 - CBO включен в полном виде всегда

Changelog category

  • New feature

Additional information

...

@pavelvelikhov pavelvelikhov requested review from a team as code owners August 27, 2024 14:46
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 27, 2024 •
edited
Loading

⚪ 2024-08-27 14:48:12 UTC Pre-commit check for 00dcb4b has started.
⚪ 2024-08-27 14:51:35 UTC Check linux-x86_64-release-asan is running...
⚫ 2024-08-27 15:00:13 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 27, 2024 •
edited
Loading

⚪ 2024-08-27 14:50:23 UTC Pre-commit check for 00dcb4b has started.
⚪ 2024-08-27 14:53:37 UTC Check linux-x86_64-relwithdebinfo is running...
⚫ 2024-08-27 15:00:12 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 27, 2024 •
edited
Loading

⚪ 2024-08-27 14:50:27 UTC Pre-commit check for 00dcb4b has started.
⚪ 2024-08-27 14:53:38 UTC Check linux-x86_64-release-clang14 is running...
⚫ 2024-08-27 15:00:11 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 27, 2024 •
edited
Loading

⚪ 2024-08-27 15:01:32 UTC Pre-commit check for d959e85 has started.
⚪ 2024-08-27 15:04:44 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-27 16:38:15 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
77604 63891 0 7 13684 22

⚪ 2024-08-27 16:46:23 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟡 2024-08-27 16:56:24 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
31 (only retried tests) 17 0 2 1 11

⚪ 2024-08-27 16:56:32 UTC Failed tests rerun (try 3) linux-x86_64-relwithdebinfo is running...
🔴 2024-08-27 17:03:51 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
18 (only retried tests) 5 0 2 0 11

🟢 2024-08-27 17:03:58 UTC Build successful.
🟢 2024-08-27 17:04:36 UTC ydbd size 8.2 GiB changed* by +13.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 4774e20 merge: d959e85 diff diff %
ydbd size 8 799 879 784 Bytes 8 799 893 824 Bytes +13.7 KiB +0.000%
ydbd stripped size 479 835 592 Bytes 479 835 784 Bytes +192 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 27, 2024 •
edited
Loading

⚪ 2024-08-27 15:01:37 UTC Pre-commit check for d959e85 has started.
⚪ 2024-08-27 15:04:52 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-27 17:15:57 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15501 15389 0 35 44 33

🟢 2024-08-27 17:17:17 UTC Build successful.
🟢 2024-08-27 17:17:50 UTC ydbd size 5.5 GiB changed* by +21.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 4774e20 merge: d959e85 diff diff %
ydbd size 5 928 702 608 Bytes 5 928 725 008 Bytes +21.9 KiB +0.000%
ydbd stripped size 1 489 038 672 Bytes 1 489 049 616 Bytes +10.7 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 27, 2024 •
edited
Loading

⚪ 2024-08-27 15:02:17 UTC Pre-commit check for d959e85 has started.
⚪ 2024-08-27 15:05:19 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-27 15:41:52 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 27, 2024 •
edited
Loading

⚪ 2024-08-27 20:55:07 UTC Pre-commit check for fc3b6c0 has started.
⚪ 2024-08-27 20:59:17 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-27 22:54:01 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15504 15428 0 23 34 19

🟢 2024-08-27 22:55:21 UTC Build successful.
🟢 2024-08-27 22:55:50 UTC ydbd size 5.5 GiB changed* by +17.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: d1d3fd7 merge: fc3b6c0 diff diff %
ydbd size 5 928 793 824 Bytes 5 928 812 120 Bytes +17.9 KiB +0.000%
ydbd stripped size 1 489 078 288 Bytes 1 489 085 136 Bytes +6.7 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 27, 2024 •
edited
Loading

⚪ 2024-08-27 20:57:27 UTC Pre-commit check for fc3b6c0 has started.
⚪ 2024-08-27 21:00:34 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-27 22:36:22 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
77617 63901 0 5 13689 22

⚪ 2024-08-27 22:43:50 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-27 22:53:55 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
33 (only retried tests) 21 0 0 0 12

🟢 2024-08-27 22:54:01 UTC Build successful.
🟢 2024-08-27 22:54:35 UTC ydbd size 8.2 GiB changed* by +13.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: d1d3fd7 merge: fc3b6c0 diff diff %
ydbd size 8 799 952 464 Bytes 8 799 966 520 Bytes +13.7 KiB +0.000%
ydbd stripped size 479 846 728 Bytes 479 846 920 Bytes +192 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 27, 2024 •
edited
Loading

⚪ 2024-08-27 20:58:01 UTC Pre-commit check for fc3b6c0 has started.
⚪ 2024-08-27 21:01:15 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-27 21:46:21 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 28, 2024 •
edited
Loading

⚪ 2024-08-28 10:50:11 UTC Pre-commit check for dcba4a1 has started.
⚪ 2024-08-28 10:53:24 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-28 11:30:20 UTC Build successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 28, 2024 •
edited
Loading

⚪ 2024-08-28 10:50:41 UTC Pre-commit check for dcba4a1 has started.
⚪ 2024-08-28 10:54:46 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-28 12:48:14 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15507 15434 0 14 38 21

🟢 2024-08-28 12:49:37 UTC Build successful.
🟢 2024-08-28 12:50:06 UTC ydbd size 5.5 GiB changed* by +21.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 22f58dc merge: dcba4a1 diff diff %
ydbd size 5 929 468 480 Bytes 5 929 490 880 Bytes +21.9 KiB +0.000%
ydbd stripped size 1 489 195 408 Bytes 1 489 206 352 Bytes +10.7 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 28, 2024 •
edited
Loading

⚪ 2024-08-28 10:50:48 UTC Pre-commit check for dcba4a1 has started.
⚪ 2024-08-28 10:54:00 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-28 12:33:28 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
77616 63908 0 4 13684 20

⚪ 2024-08-28 12:41:06 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-28 12:48:34 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
27 (only retried tests) 14 0 0 1 12

🟢 2024-08-28 12:48:42 UTC Build successful.
🟢 2024-08-28 12:49:16 UTC ydbd size 8.2 GiB changed* by +13.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 22f58dc merge: dcba4a1 diff diff %
ydbd size 8 800 972 216 Bytes 8 800 986 272 Bytes +13.7 KiB +0.000%
ydbd stripped size 479 883 592 Bytes 479 883 784 Bytes +192 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@pavelvelikhov pavelvelikhov enabled auto-merge (squash) August 28, 2024 12:27
resetius
resetius reviewed Aug 28, 2024
View reviewed changes
ydb/core/kqp/opt/logical/kqp_opt_cbo.cpp
@@ -148,7 +148,7 @@ bool TKqpProviderContext::IsJoinApplicable(const std::shared_ptr<IBaseOptimizerN

switch( joinAlgo ) {
case EJoinAlgoType::LookupJoin:
if ((OptLevel >= 2) && (left->Stats->Nrows > 1000)) {
if ((OptLevel != 3) && (left->Stats->Nrows > 1000)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

почему не равно? для 2 и 4 будет false?

Copy link
Collaborator Author

@pavelvelikhov pavelvelikhov Aug 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Да, так правильно сейчас

Уровень 3 - специальный, он форсирует lookup joins всегда, если они в принципе возможны. На других уровнях проверяем условие Nrows

Кстати с новыми кост функциями эта проверка на 1000 записей уйдет

@pavelvelikhov pavelvelikhov merged commit bfadc55 into ydb-platform:main Aug 28, 2024
11 of 13 checks passed
rekby pushed a commit to rekby-forks/ydb that referenced this pull request Aug 28, 2024
@pavelvelikhov @rekby
Added another cost based optimization level (ydb-platform#8348)
5b6f835
@niksaveliev niksaveliev mentioned this pull request Aug 29, 2024
Merged
@siarheivesialou siarheivesialou mentioned this pull request Aug 30, 2024
Merged
@GrigoriyPA GrigoriyPA mentioned this pull request Aug 30, 2024
Closed
@siarheivesialou siarheivesialou mentioned this pull request Aug 30, 2024
Merged
stanislav-shchetinin pushed a commit to stanislav-shchetinin/ydb that referenced this pull request Aug 30, 2024
@pavelvelikhov @stanislav-shchetinin
Added another cost based optimization level (ydb-platform#8348)
2753e57
@siarheivesialou siarheivesialou mentioned this pull request Sep 4, 2024
Merged
This was referenced Sep 4, 2024
Merged
Merged
Merged
pavelvelikhov added a commit to pavelvelikhov/ydb that referenced this pull request Sep 5, 2024
@pavelvelikhov
Added another cost based optimization level (ydb-platform#8348)
f91d492
This was referenced Sep 9, 2024
Merged
Merged
@StekPerepolnen StekPerepolnen mentioned this pull request Sep 17, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@resetius resetius resetius approved these changes

@spuchin spuchin spuchin approved these changes

Assignees
No one assigned
Labels
new-feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@pavelvelikhov @resetius @spuchin