Skip to content

Pass compacted pages directly to Shared Cache #11779

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Nov 27, 2024

Conversation

kunga
Copy link
Member

@kunga kunga commented Nov 20, 2024

Changelog entry

...

Changelog category

  • Not for changelog (changelog entry is not required)

Additional information

As a first step of removing Private Cache Share Body logic, stop using it for compacted pages

@kunga kunga self-assigned this Nov 20, 2024
@kunga kunga mentioned this pull request Nov 20, 2024
30 tasks
Copy link

github-actions bot commented Nov 20, 2024

2024-11-20 10:03:58 UTC Pre-commit check linux-x86_64-relwithdebinfo for e755d34 has started.
2024-11-20 10:04:03 UTC Artifacts will be uploaded here
2024-11-20 10:07:16 UTC ya make is running...
🔴 2024-11-20 10:10:18 UTC Build failed, see the logs. Also see fail summary

Copy link

github-actions bot commented Nov 20, 2024

2024-11-20 10:06:10 UTC Pre-commit check linux-x86_64-release-asan for e755d34 has started.
2024-11-20 10:06:21 UTC Artifacts will be uploaded here
2024-11-20 10:09:20 UTC ya make is running...
🔴 2024-11-20 10:12:28 UTC Build failed, see the logs. Also see fail summary

Copy link

github-actions bot commented Nov 21, 2024

2024-11-21 17:32:04 UTC Pre-commit check linux-x86_64-relwithdebinfo for df28f7c has started.
2024-11-21 17:32:15 UTC Artifacts will be uploaded here
2024-11-21 17:35:16 UTC ya make is running...
🔴 2024-11-21 18:38:04 UTC Too many tests failed, NOT going to retry
🔴 2024-11-21 18:40:39 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15754 12424 0 1145 2048 137

🟢 2024-11-21 18:42:45 UTC Build successful.
🟢 2024-11-21 18:43:10 UTC ydbd size 2.5 GiB changed* by +28.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 84dc18a merge: df28f7c diff diff %
ydbd size 2 689 793 064 Bytes 2 689 822 328 Bytes +28.6 KiB +0.001%
ydbd stripped size 481 860 560 Bytes 481 871 696 Bytes +10.9 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Nov 21, 2024

2024-11-21 17:33:09 UTC Pre-commit check linux-x86_64-release-asan for df28f7c has started.
2024-11-21 17:33:20 UTC Artifacts will be uploaded here
2024-11-21 17:36:16 UTC ya make is running...
🔴 2024-11-21 18:42:20 UTC Too many tests failed, NOT going to retry
🟡 2024-11-21 18:43:41 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9290 8382 0 620 233 55

🟢 2024-11-21 18:44:32 UTC Build successful.
🟢 2024-11-21 18:45:06 UTC ydbd size 4.9 GiB changed* by +73.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: b475488 merge: df28f7c diff diff %
ydbd size 5 281 208 376 Bytes 5 281 283 936 Bytes +73.8 KiB +0.001%
ydbd stripped size 1 358 582 000 Bytes 1 358 604 592 Bytes +22.1 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Nov 22, 2024

2024-11-22 14:34:57 UTC Pre-commit check linux-x86_64-relwithdebinfo for b50f3c6 has started.
2024-11-22 14:35:09 UTC Artifacts will be uploaded here
2024-11-22 14:36:54 UTC Check cancelled

Copy link

github-actions bot commented Nov 22, 2024

2024-11-22 14:35:02 UTC Pre-commit check linux-x86_64-release-asan for b50f3c6 has started.
2024-11-22 14:35:13 UTC Artifacts will be uploaded here
2024-11-22 14:36:53 UTC Check cancelled

Copy link

github-actions bot commented Nov 22, 2024

2024-11-22 14:40:06 UTC Pre-commit check linux-x86_64-relwithdebinfo for e20e0be has started.
2024-11-22 14:40:19 UTC Artifacts will be uploaded here
2024-11-22 14:43:17 UTC ya make is running...
2024-11-22 15:10:24 UTC Check cancelled

Copy link

github-actions bot commented Nov 22, 2024

2024-11-22 14:42:41 UTC Pre-commit check linux-x86_64-release-asan for e20e0be has started.
2024-11-22 14:42:53 UTC Artifacts will be uploaded here
2024-11-22 14:45:59 UTC ya make is running...
2024-11-22 15:10:34 UTC Check cancelled

Copy link

github-actions bot commented Nov 22, 2024

2024-11-22 15:14:35 UTC Pre-commit check linux-x86_64-release-asan for 387c7d8 has started.
2024-11-22 15:14:59 UTC Artifacts will be uploaded here
2024-11-22 15:17:58 UTC ya make is running...
2024-11-22 15:19:18 UTC Check cancelled

Copy link

github-actions bot commented Nov 22, 2024

2024-11-22 15:14:54 UTC Pre-commit check linux-x86_64-relwithdebinfo for 387c7d8 has started.
2024-11-22 15:15:36 UTC Artifacts will be uploaded here
2024-11-22 15:19:16 UTC Check cancelled

Copy link

github-actions bot commented Nov 22, 2024

2024-11-22 15:20:06 UTC Pre-commit check linux-x86_64-relwithdebinfo for 9bf3d49 has started.
2024-11-22 15:20:10 UTC Artifacts will be uploaded here
2024-11-22 15:23:15 UTC ya make is running...
🟡 2024-11-22 16:15:41 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15716 14184 0 9 1410 113

2024-11-22 16:17:04 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-11-22 16:28:44 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
121 (only retried tests) 10 0 2 0 109

2024-11-22 16:28:52 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-11-22 16:40:31 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
111 (only retried tests) 2 0 1 0 108

🟢 2024-11-22 16:40:40 UTC Build successful.
🟢 2024-11-22 16:41:00 UTC ydbd size 2.5 GiB changed* by +43.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 198242d merge: 9bf3d49 diff diff %
ydbd size 2 691 739 688 Bytes 2 691 784 648 Bytes +43.9 KiB +0.002%
ydbd stripped size 482 183 088 Bytes 482 192 496 Bytes +9.2 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Nov 22, 2024

2024-11-22 15:21:23 UTC Pre-commit check linux-x86_64-release-asan for 9bf3d49 has started.
2024-11-22 15:21:39 UTC Artifacts will be uploaded here
2024-11-22 15:24:35 UTC ya make is running...
🟡 2024-11-22 16:37:48 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9308 9030 0 75 146 57

🟢 2024-11-22 16:38:36 UTC Build successful.
🟡 2024-11-22 16:39:05 UTC ydbd size 4.9 GiB changed* by +108.1 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 198242d merge: 9bf3d49 diff diff %
ydbd size 5 285 188 648 Bytes 5 285 299 368 Bytes +108.1 KiB +0.002%
ydbd stripped size 1 359 580 208 Bytes 1 359 612 880 Bytes +31.9 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Nov 25, 2024

2024-11-25 10:46:23 UTC Pre-commit check linux-x86_64-relwithdebinfo for b0b8730 has started.
2024-11-25 10:46:35 UTC Artifacts will be uploaded here
2024-11-25 10:48:50 UTC Check cancelled

@@ -0,0 +1,68 @@
#pragma once
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Вынос кода из shared_sausagecache.cpp

Copy link

github-actions bot commented Nov 25, 2024

2024-11-25 10:50:29 UTC Pre-commit check linux-x86_64-release-asan for 5ac9544 has started.
2024-11-25 10:50:41 UTC Artifacts will be uploaded here
2024-11-25 10:53:52 UTC ya make is running...
🟡 2024-11-25 12:25:48 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10499 10423 0 17 9 50

🟢 2024-11-25 12:26:42 UTC Build successful.
🟡 2024-11-25 12:27:09 UTC ydbd size 4.9 GiB changed* by +126.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: afbc266 merge: 5ac9544 diff diff %
ydbd size 5 284 319 504 Bytes 5 284 448 824 Bytes +126.3 KiB +0.002%
ydbd stripped size 1 359 968 560 Bytes 1 360 006 768 Bytes +37.3 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Nov 25, 2024

2024-11-25 10:51:55 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5ac9544 has started.
2024-11-25 10:52:05 UTC Artifacts will be uploaded here
2024-11-25 10:55:16 UTC ya make is running...
🟡 2024-11-25 12:18:43 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
26256 23422 0 2 2719 113

2024-11-25 12:20:51 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-11-25 12:32:13 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
114 (only retried tests) 7 0 1 0 106

2024-11-25 12:32:21 UTC ya make is running... (failed tests rerun, try 3)
🟢 2024-11-25 12:41:47 UTC ydbd size 2.5 GiB changed* by +52.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: afbc266 merge: 5ac9544 diff diff %
ydbd size 2 690 053 656 Bytes 2 690 107 352 Bytes +52.4 KiB +0.002%
ydbd stripped size 482 185 488 Bytes 482 196 112 Bytes +10.4 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation
2024-11-25 12:41:48 UTC Check cancelled

Copy link

github-actions bot commented Nov 25, 2024

2024-11-25 12:43:48 UTC Pre-commit check linux-x86_64-relwithdebinfo for e658f95 has started.
2024-11-25 12:43:58 UTC Artifacts will be uploaded here
2024-11-25 12:47:09 UTC ya make is running...
🟢 2024-11-25 14:17:25 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
26256 23424 0 0 2719 113

🟢 2024-11-25 14:19:31 UTC Build successful.
🟢 2024-11-25 14:19:48 UTC ydbd size 2.5 GiB changed* by +83.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 346aedc merge: e658f95 diff diff %
ydbd size 2 690 056 568 Bytes 2 690 142 048 Bytes +83.5 KiB +0.003%
ydbd stripped size 482 185 488 Bytes 482 196 496 Bytes +10.8 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Nov 25, 2024

2024-11-25 12:43:54 UTC Pre-commit check linux-x86_64-release-asan for e658f95 has started.
2024-11-25 12:44:05 UTC Artifacts will be uploaded here
2024-11-25 12:47:18 UTC ya make is running...
🟡 2024-11-25 14:26:22 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10496 10425 0 16 8 47

🟢 2024-11-25 14:27:14 UTC Build successful.
🟡 2024-11-25 14:27:40 UTC ydbd size 4.9 GiB changed* by +187.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 346aedc merge: e658f95 diff diff %
ydbd size 5 284 326 944 Bytes 5 284 518 712 Bytes +187.3 KiB +0.004%
ydbd stripped size 1 359 968 560 Bytes 1 360 008 176 Bytes +38.7 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@kunga kunga requested a review from snaury November 25, 2024 12:44
@kunga kunga marked this pull request as ready for review November 25, 2024 12:45
const TLogoBlobID metaId = pageCollection.Label();

Y_ABORT_UNLESS(metaId);
Y_ABORT_IF(Collections.contains(metaId), "Only new collections can save compacted pages");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Сомнительная верифайка, там точно не может быть гонки, когда эту мету уже успели зарегистрировать до того как этот TEvSaveCompactedPages отправился? К сожалению не вижу как это сообщение используется.

Copy link
Member Author

@kunga kunga Nov 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Отправка вот https://github.com/ydb-platform/ydb/pull/11779/files#diff-fda9484ae1850f6173959c26bb38ec82f2e846b002e247b879051ee9c3546c9bR340 в момент когда формируются результаты компакшена, таблетка их получит уже потом вот тут

AddCachesOfBundle(stage->PartView);
когда будет кэши заполнять, и в этот момент отправит TEvAttach

То есть Send(TEvSaveCompactedPages) от TOpsCompact точно будет раньше чем Send(TEvAttach) таблетки, так что как будто и сообщения в том же порядке должны приходить

Copy link

github-actions bot commented Nov 26, 2024

2024-11-26 15:56:31 UTC Pre-commit check linux-x86_64-relwithdebinfo for 0b5b727 has started.
2024-11-26 15:57:05 UTC Artifacts will be uploaded here
2024-11-26 16:01:09 UTC ya make is running...
🟡 2024-11-26 17:14:59 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
26265 23423 0 4 2719 119

2024-11-26 17:17:11 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-11-26 17:28:48 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
121 (only retried tests) 12 0 0 0 109

🟢 2024-11-26 17:28:55 UTC Build successful.
🟢 2024-11-26 17:29:15 UTC ydbd size 2.5 GiB changed* by +48.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 205cf3a merge: 0b5b727 diff diff %
ydbd size 2 694 602 432 Bytes 2 694 651 840 Bytes +48.2 KiB +0.002%
ydbd stripped size 482 598 928 Bytes 482 608 976 Bytes +9.8 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Nov 26, 2024

2024-11-26 15:57:57 UTC Pre-commit check linux-x86_64-release-asan for 0b5b727 has started.
2024-11-26 15:58:09 UTC Artifacts will be uploaded here
2024-11-26 16:01:27 UTC ya make is running...
🟡 2024-11-26 17:27:06 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10505 10432 0 17 9 47

🟢 2024-11-26 17:28:00 UTC Build successful.
🟡 2024-11-26 17:28:29 UTC ydbd size 4.9 GiB changed* by +113.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 205cf3a merge: 0b5b727 diff diff %
ydbd size 5 293 849 152 Bytes 5 293 965 480 Bytes +113.6 KiB +0.002%
ydbd stripped size 1 361 405 648 Bytes 1 361 438 992 Bytes +32.6 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@kunga kunga requested a review from snaury November 26, 2024 15:59
@kunga kunga merged commit d6aa89c into ydb-platform:main Nov 27, 2024
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants