Skip to content

Propagate wide-block through channels #6096

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Jul 25, 2024
Merged

Conversation

abyss7
Copy link
Collaborator

@abyss7 abyss7 commented Jun 29, 2024

Implement ad-hoc peephole optimization that propagates "WideFromBlocks()" nodes through wide channels - on topologically-sorted stages inside transaction.

Copy link

github-actions bot commented Jun 29, 2024

2024-06-29 09:24:53 UTC Pre-commit check for d560814 has started.
2024-06-29 09:27:19 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-06-29 09:32:15 UTC Build successful.

Copy link

github-actions bot commented Jun 29, 2024

2024-06-29 09:25:14 UTC Pre-commit check for d560814 has started.
2024-06-29 09:27:44 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-06-29 10:07:03 UTC Build successful.
2024-06-29 10:07:21 UTC Tests are running...

Copy link

github-actions bot commented Jun 29, 2024

2024-06-29 09:43:12 UTC Pre-commit check for d560814 has started.
2024-06-29 09:45:39 UTC Build linux-x86_64-release-asan is running...
🟢 2024-06-29 10:09:54 UTC Build successful.
2024-06-29 10:10:17 UTC Tests are running...

Copy link

github-actions bot commented Jun 30, 2024

2024-06-30 08:38:35 UTC Pre-commit check for 0af5f99 has started.
2024-06-30 08:41:00 UTC Build linux-x86_64-release-asan is running...
🟢 2024-06-30 09:05:28 UTC Build successful.
2024-06-30 09:05:56 UTC Tests are running...
🔴 2024-06-30 11:07:03 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9661 9080 0 86 320 175

🟡 2024-06-30 11:07:56 UTC ydbd size 5.5 GiB changed* by +776.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: c56118c merge: 0af5f99 diff diff %
ydbd size 5 902 121 360 Bytes 5 902 916 920 Bytes +776.9 KiB +0.013%
ydbd stripped size 1 269 085 448 Bytes 1 269 226 056 Bytes +137.3 KiB +0.011%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jun 30, 2024

2024-06-30 09:08:11 UTC Pre-commit check for 0af5f99 has started.
2024-06-30 09:10:36 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-06-30 09:15:25 UTC Build successful.

Copy link

github-actions bot commented Jun 30, 2024

2024-06-30 09:17:02 UTC Pre-commit check for 0af5f99 has started.
2024-06-30 09:19:27 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-06-30 09:58:25 UTC Build successful.
2024-06-30 09:58:46 UTC Tests are running...
🔴 2024-06-30 11:30:46 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13764 12434 0 13 1305 12

🟡 2024-06-30 11:31:24 UTC ydbd size 8.4 GiB changed* by +1.0 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: c56118c merge: 0af5f99 diff diff %
ydbd size 8 999 169 672 Bytes 9 000 268 336 Bytes +1.0 MiB +0.012%
ydbd stripped size 492 128 712 Bytes 492 165 448 Bytes +35.9 KiB +0.007%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 1, 2024

2024-07-01 09:32:36 UTC Pre-commit check for b8903b9 has started.
2024-07-01 09:35:09 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-07-01 09:40:37 UTC Build successful.

Copy link

github-actions bot commented Jul 1, 2024

2024-07-01 09:39:04 UTC Pre-commit check for b8903b9 has started.
2024-07-01 09:41:31 UTC Build linux-x86_64-release-asan is running...
🟢 2024-07-01 10:04:52 UTC Build successful.
2024-07-01 10:05:06 UTC Tests are running...
🔴 2024-07-01 12:12:02 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9711 8577 0 501 443 190

🟡 2024-07-01 12:15:23 UTC ydbd size 5.4 GiB changed* by +757.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: a874417 merge: b8903b9 diff diff %
ydbd size 5 831 012 856 Bytes 5 831 788 248 Bytes +757.2 KiB +0.013%
ydbd stripped size 1 251 038 120 Bytes 1 251 175 464 Bytes +134.1 KiB +0.011%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 1, 2024

2024-07-01 10:25:38 UTC Pre-commit check for b8903b9 has started.
2024-07-01 10:28:34 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-07-01 11:06:34 UTC Build successful.
2024-07-01 11:06:55 UTC Tests are running...
🔴 2024-07-01 12:44:51 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13864 11818 0 685 1326 35

🟡 2024-07-01 12:46:09 UTC ydbd size 8.3 GiB changed* by +1.0 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: a874417 merge: b8903b9 diff diff %
ydbd size 8 953 008 608 Bytes 8 954 090 224 Bytes +1.0 MiB +0.012%
ydbd stripped size 488 238 568 Bytes 488 270 568 Bytes +31.2 KiB +0.007%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 5, 2024

2024-07-05 13:27:22 UTC Pre-commit check for c5f4c54 has started.
2024-07-05 13:30:24 UTC Build linux-x86_64-release-asan is running...
🟢 2024-07-05 13:52:43 UTC Build successful.
2024-07-05 13:52:53 UTC Tests are running...
🔴 2024-07-05 15:46:45 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9749 9168 0 83 325 173

🟡 2024-07-05 15:47:26 UTC ydbd size 5.2 GiB changed* by +1.1 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 5fc09ca merge: c5f4c54 diff diff %
ydbd size 5 615 862 464 Bytes 5 617 064 440 Bytes +1.1 MiB +0.021%
ydbd stripped size 1 211 107 272 Bytes 1 211 349 832 Bytes +236.9 KiB +0.020%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 5, 2024

2024-07-05 13:52:08 UTC Pre-commit check for c5f4c54 has started.
2024-07-05 13:54:36 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-07-05 13:59:12 UTC Build successful.

Copy link

github-actions bot commented Jul 5, 2024

2024-07-05 13:56:46 UTC Pre-commit check for c5f4c54 has started.
2024-07-05 13:59:21 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-07-05 14:37:40 UTC Build successful.
2024-07-05 14:37:52 UTC Tests are running...
🔴 2024-07-05 16:05:26 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13860 12543 0 2 1306 9

🟡 2024-07-05 16:05:57 UTC ydbd size 8.1 GiB changed* by +1.2 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 5fc09ca merge: c5f4c54 diff diff %
ydbd size 8 726 129 816 Bytes 8 727 439 560 Bytes +1.2 MiB +0.015%
ydbd stripped size 477 144 232 Bytes 477 198 248 Bytes +52.8 KiB +0.011%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 7, 2024

2024-07-07 18:45:10 UTC Pre-commit check for 221e394 has started.
2024-07-07 18:47:37 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-07-07 18:52:52 UTC Build successful.

Copy link

github-actions bot commented Jul 7, 2024

2024-07-07 18:45:22 UTC Pre-commit check for 221e394 has started.
2024-07-07 18:47:54 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-07-07 19:24:28 UTC Build successful.
2024-07-07 19:24:42 UTC Tests are running...
🟢 2024-07-07 20:47:03 UTC Tests successful.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13868 12550 0 0 1307 11

🟡 2024-07-07 20:47:52 UTC ydbd size 8.1 GiB changed* by +1.1 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 77685b3 merge: 221e394 diff diff %
ydbd size 8 721 673 992 Bytes 8 722 871 224 Bytes +1.1 MiB +0.014%
ydbd stripped size 476 619 640 Bytes 476 666 488 Bytes +45.8 KiB +0.010%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 7, 2024

2024-07-07 18:45:22 UTC Pre-commit check for 221e394 has started.
2024-07-07 18:47:55 UTC Build linux-x86_64-release-asan is running...
🟢 2024-07-07 19:11:04 UTC Build successful.
2024-07-07 19:11:18 UTC Tests are running...
🔴 2024-07-07 21:10:03 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9753 9178 0 79 324 172

🟡 2024-07-07 21:10:51 UTC ydbd size 5.2 GiB changed* by +1.1 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 77685b3 merge: 221e394 diff diff %
ydbd size 5 614 060 320 Bytes 5 615 172 760 Bytes +1.1 MiB +0.020%
ydbd stripped size 1 209 974 104 Bytes 1 210 205 272 Bytes +225.8 KiB +0.019%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

// Sort stages in topological order by their inputs, so that we optimize the ones without inputs first.
TVector<TDqPhyStage> topSortedStages;
topSortedStages.reserve(tx.Stages().Size());
{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will be better to move topsort to separate function

}
}

if (needRebuild) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we don't need needRebuild var

Copy link

github-actions bot commented Jul 8, 2024

2024-07-08 15:33:48 UTC Pre-commit check for 3195bad has started.
2024-07-08 15:36:31 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-07-08 16:06:20 UTC Build successful.

Copy link

github-actions bot commented Jul 8, 2024

2024-07-08 19:34:48 UTC Pre-commit check for 3195bad has started.
2024-07-08 19:37:32 UTC Build linux-x86_64-release-asan is running...
🟢 2024-07-08 19:59:10 UTC Build successful.
2024-07-08 19:59:22 UTC Tests are running...

Copy link

github-actions bot commented Jul 8, 2024

2024-07-08 19:14:08 UTC Pre-commit check for 3195bad has started.
2024-07-08 19:17:01 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-07-08 19:53:46 UTC Build successful.
2024-07-08 19:54:09 UTC Tests are running...
🔴 2024-07-08 23:46:40 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39874 30257 0 2444 6744 429

🟡 2024-07-08 23:53:39 UTC ydbd size 8.1 GiB changed* by +1.1 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: d65d7ea merge: 3195bad diff diff %
ydbd size 8 709 604 056 Bytes 8 710 801 472 Bytes +1.1 MiB +0.014%
ydbd stripped size 475 601 016 Bytes 475 647 864 Bytes +45.8 KiB +0.010%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@abyss7 abyss7 requested a review from a team as a code owner July 9, 2024 11:23
Copy link

github-actions bot commented Jul 9, 2024

2024-07-09 11:51:50 UTC Pre-commit check for fdbaf2a has started.
2024-07-09 11:54:35 UTC Build linux-x86_64-release-clang14 is running...
🔴 2024-07-09 12:01:13 UTC Build failed. see the build logs.

Copy link

github-actions bot commented Jul 9, 2024

2024-07-09 12:14:11 UTC Pre-commit check for fdbaf2a has started.
2024-07-09 12:16:55 UTC Build linux-x86_64-release-asan is running...
🔴 2024-07-09 12:24:18 UTC Build failed. see the build logs.
🔴 2024-07-09 12:24:19 UTC Tests run skipped.

Copy link

github-actions bot commented Jul 9, 2024

2024-07-09 12:35:09 UTC Pre-commit check for 2a42c7d has started.
2024-07-09 12:38:25 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-07-09 13:08:19 UTC Build successful.

Copy link

github-actions bot commented Jul 9, 2024

2024-07-09 12:36:37 UTC Pre-commit check for 2a42c7d has started.
2024-07-09 12:39:23 UTC Build linux-x86_64-release-asan is running...
🟢 2024-07-09 13:15:38 UTC Build successful.
2024-07-09 13:15:53 UTC Tests are running...

Copy link

github-actions bot commented Jul 9, 2024

2024-07-09 14:11:26 UTC Pre-commit check for 2a42c7d has started.
2024-07-09 14:14:16 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-07-09 15:08:19 UTC Build successful.
2024-07-09 15:08:36 UTC Tests are running...

Copy link

github-actions bot commented Jul 15, 2024

2024-07-15 14:22:19 UTC Pre-commit check for cf83b7b has started.
2024-07-15 14:23:40 UTC Check cancelled

Copy link

github-actions bot commented Jul 15, 2024

2024-07-15 14:33:30 UTC Pre-commit check for cfbdca3 has started.
2024-07-15 14:36:48 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🟢 2024-07-15 16:11:48 UTC Build successful.
🔴 2024-07-15 16:16:47 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13697 12364 0 7 1309 17

🟢 2024-07-15 16:17:32 UTC ydbd size 8.1 GiB changed* by -6.7 MiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 85d4495 merge: cfbdca3 diff diff %
ydbd size 8 691 124 032 Bytes 8 684 050 888 Bytes -6.7 MiB -0.081%
ydbd stripped size 473 665 904 Bytes 473 291 056 Bytes -366.1 KiB -0.079%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 15, 2024

2024-07-15 14:34:47 UTC Pre-commit check for cfbdca3 has started.
2024-07-15 14:37:58 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-15 14:43:09 UTC Build successful.

Copy link

github-actions bot commented Jul 15, 2024

2024-07-15 14:48:50 UTC Pre-commit check for cfbdca3 has started.
2024-07-15 14:51:30 UTC Build+Tests linux-x86_64-release-asan is running...
🟢 2024-07-15 16:58:29 UTC Build successful.
🔴 2024-07-15 17:00:22 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9584 8994 0 87 325 178

🟡 2024-07-15 17:00:59 UTC ydbd size 5.2 GiB changed* by +1.1 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: ab051d7 merge: cfbdca3 diff diff %
ydbd size 5 567 530 472 Bytes 5 568 635 512 Bytes +1.1 MiB +0.020%
ydbd stripped size 1 199 874 160 Bytes 1 200 095 728 Bytes +216.4 KiB +0.018%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 17, 2024

2024-07-17 13:37:41 UTC Pre-commit check for 3ed0821 has started.
2024-07-17 13:40:20 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-17 13:45:27 UTC Build successful.

Copy link

github-actions bot commented Jul 17, 2024

2024-07-17 13:39:52 UTC Pre-commit check for 3ed0821 has started.
2024-07-17 13:42:34 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🟢 2024-07-17 15:11:12 UTC Build successful.
🔴 2024-07-17 15:12:51 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13698 12367 0 8 1309 14

🟡 2024-07-17 15:13:33 UTC ydbd size 8.0 GiB changed* by +1.1 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 02f9150 merge: 3ed0821 diff diff %
ydbd size 8 641 949 216 Bytes 8 643 065 144 Bytes +1.1 MiB +0.013%
ydbd stripped size 470 498 400 Bytes 470 543 200 Bytes +43.8 KiB +0.010%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 17, 2024

2024-07-17 13:40:10 UTC Pre-commit check for 3ed0821 has started.
2024-07-17 13:42:46 UTC Build+Tests linux-x86_64-release-asan is running...
🟢 2024-07-17 15:42:37 UTC Build successful.
🔴 2024-07-17 15:44:47 UTC Some tests failed, follow the links below.

Test history | Test log

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9589 9008 0 82 325 174

🟡 2024-07-17 15:45:35 UTC ydbd size 5.2 GiB changed* by +929.7 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 02f9150 merge: 3ed0821 diff diff %
ydbd size 5 546 475 544 Bytes 5 547 427 528 Bytes +929.7 KiB +0.017%
ydbd stripped size 1 195 222 000 Bytes 1 195 400 944 Bytes +174.8 KiB +0.015%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 21, 2024

2024-07-21 08:30:21 UTC Pre-commit check for f238374 has started.
2024-07-21 08:33:09 UTC Build+Tests linux-x86_64-release-asan is running...
🟢 2024-07-21 10:36:32 UTC Build successful.
🔴 2024-07-21 10:40:31 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9620 9022 0 95 330 173

🟡 2024-07-21 10:41:30 UTC ydbd size 5.2 GiB changed* by +942.1 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 5e95651 merge: f238374 diff diff %
ydbd size 5 556 133 632 Bytes 5 557 098 352 Bytes +942.1 KiB +0.017%
ydbd stripped size 1 197 007 984 Bytes 1 197 189 488 Bytes +177.2 KiB +0.015%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 21, 2024

2024-07-21 08:30:27 UTC Pre-commit check for f238374 has started.
2024-07-21 08:33:06 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-21 08:38:54 UTC Build successful.

Copy link

github-actions bot commented Jul 21, 2024

2024-07-21 08:30:29 UTC Pre-commit check for f238374 has started.
2024-07-21 08:33:09 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🟢 2024-07-21 10:03:08 UTC Build successful.
🔴 2024-07-21 10:04:44 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13730 12403 0 4 1313 10

🔴 2024-07-21 10:05:20 UTC ydbd size 8.1 GiB changed* by +2.1 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 1c15550 merge: f238374 diff diff %
ydbd size 8 656 109 112 Bytes 8 658 286 216 Bytes +2.1 MiB +0.025%
ydbd stripped size 471 115 872 Bytes 471 181 920 Bytes +64.5 KiB +0.014%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@gridnevvvit gridnevvvit self-requested a review July 25, 2024 13:04
@abyss7 abyss7 merged commit 4d1e99e into ydb-platform:main Jul 25, 2024
9 of 13 checks passed
abyss7 added a commit to abyss7/ydb that referenced this pull request Sep 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants