The race between `TEvProposeTransaction` and `TEvLockStatus` #8517

Alek5andr-Kotov · 2024-08-30T09:54:17Z

Changelog entry

The entry for WriteId was not saved if there was an empty list of partitions in TTxWriteInfo. As a result, after restoring the tablet, Y_VERIFY was triggered.

Changelog category

Bugfix
Not for changelog (changelog entry is not required)

Additional information

...

github-actions · 2024-08-30T09:55:36Z

⚪ 2024-08-30 09:55:35 UTC Pre-commit check linux-x86_64-relwithdebinfo for c2ccab9 has started.
⚪ 2024-08-30 09:58:27 UTC ya make is running...
🟡 2024-08-30 11:01:03 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14652	13230	0	47	1356	19

⚪ 2024-08-30 11:02:18 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-08-30 11:11:14 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
74 (only retried tests)	58	0	2	0	14

⚪ 2024-08-30 11:11:22 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-08-30 11:19:22 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
23 (only retried tests)	8	0	2	0	13

🟢 2024-08-30 11:19:29 UTC Build successful.
🟢 2024-08-30 11:20:10 UTC ydbd size 8.2 GiB changed* by +19.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `dc4ea7f`	merge: `c2ccab9`	diff	diff %
ydbd size	8 820 158 464 Bytes	8 820 178 536 Bytes	+19.6 KiB	+0.000%
ydbd stripped size	480 879 688 Bytes	480 881 544 Bytes	+1.8 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-08-30T09:55:58Z

⚪ 2024-08-30 09:55:58 UTC Pre-commit check linux-x86_64-release-asan for c2ccab9 has started.
⚪ 2024-08-30 09:58:41 UTC ya make is running...
🔴 2024-08-30 11:25:28 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10037	9983	0	13	23	18

🟢 2024-08-30 11:26:20 UTC Build successful.
🟢 2024-08-30 11:26:47 UTC ydbd size 5.5 GiB changed* by +24.0 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `dc4ea7f`	merge: `c2ccab9`	diff	diff %
ydbd size	5 944 947 824 Bytes	5 944 972 360 Bytes	+24.0 KiB	+0.000%
ydbd stripped size	1 492 858 064 Bytes	1 492 863 312 Bytes	+5.1 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-08-30T09:57:34Z

⚪ 2024-08-30 09:57:33 UTC Pre-commit check linux-x86_64-release-clang14 for c2ccab9 has started.
⚪ 2024-08-30 10:00:17 UTC ya make is running...
🟢 2024-08-30 10:06:31 UTC Build successful.

Alek5andr-Kotov · 2024-08-30T10:01:19Z

@alexnick88

Транзакции удаляются по времени
https://github.com/Alek5andr-Kotov/ydb/blob/LOGBROKER-9232/ydb/core/persqueue/pq_impl.cpp#L3218
Реакция на TEvLockStatus
https://github.com/Alek5andr-Kotov/ydb/blob/LOGBROKER-9232/ydb/core/persqueue/pq_impl.cpp#L4773
https://github.com/Alek5andr-Kotov/ydb/blob/LOGBROKER-9232/ydb/core/persqueue/pq_impl.cpp#L4787
В обработчике TEvProposeTransaction проверяется WriteId
https://github.com/Alek5andr-Kotov/ydb/blob/LOGBROKER-9232/ydb/core/persqueue/pq_impl.cpp#L3354
Удалил код с Y_VERIFY

Alek5andr-Kotov · 2024-08-30T10:14:55Z

ydb/core/persqueue/pq_impl.cpp

-    Y_ABORT_UNLESS(TxWrites.contains(*writeId),
-                   "PQ %" PRIu64 ", WriteId {%" PRIu64 ", %" PRIu64 "}",
-                   TabletID(), writeId->NodeId, writeId->KeyId);
-


Удалил это Y_VERIFY

github-actions · 2024-08-30T11:50:23Z

⚪ 2024-08-30 11:50:22 UTC Pre-commit check linux-x86_64-release-clang14 for 8ec32dd has started.
⚪ 2024-08-30 11:53:06 UTC ya make is running...
🟢 2024-08-30 11:58:38 UTC Build successful.

github-actions · 2024-08-30T11:50:34Z

⚪ 2024-08-30 11:50:34 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8ec32dd has started.
⚪ 2024-08-30 11:53:19 UTC ya make is running...
🟡 2024-08-30 12:56:22 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14646	13253	0	9	1361	23

⚪ 2024-08-30 12:57:43 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-08-30 13:08:58 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
49 (only retried tests)	26	0	5	0	18

⚪ 2024-08-30 13:09:06 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-08-30 13:20:13 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
30 (only retried tests)	9	0	5	0	16

🟢 2024-08-30 13:20:21 UTC Build successful.
🟢 2024-08-30 13:21:01 UTC ydbd size 8.2 GiB changed* by +28.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `9fe138b`	merge: `8ec32dd`	diff	diff %
ydbd size	8 820 202 192 Bytes	8 820 231 144 Bytes	+28.3 KiB	+0.000%
ydbd stripped size	480 882 120 Bytes	480 884 936 Bytes	+2.8 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-08-30T11:51:49Z

⚪ 2024-08-30 11:51:49 UTC Pre-commit check linux-x86_64-release-asan for 8ec32dd has started.
⚪ 2024-08-30 11:55:34 UTC ya make is running...
🔴 2024-08-30 13:32:07 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10044	9947	0	22	48	27

🟢 2024-08-30 13:33:02 UTC Build successful.
🟢 2024-08-30 13:33:33 UTC ydbd size 5.5 GiB changed* by +33.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `9fe138b`	merge: `8ec32dd`	diff	diff %
ydbd size	5 944 988 984 Bytes	5 945 022 864 Bytes	+33.1 KiB	+0.001%
ydbd stripped size	1 492 870 544 Bytes	1 492 877 392 Bytes	+6.7 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-02T08:12:00Z

⚪ 2024-09-02 08:12:00 UTC Pre-commit check linux-x86_64-release-clang14 for 24b8935 has started.
⚪ 2024-09-02 08:15:27 UTC ya make is running...
🟢 2024-09-02 08:21:54 UTC Build successful.

github-actions · 2024-09-02T08:12:21Z

⚪ 2024-09-02 08:12:20 UTC Pre-commit check linux-x86_64-relwithdebinfo for 24b8935 has started.
⚪ 2024-09-02 08:15:08 UTC ya make is running...
🟡 2024-09-02 09:15:27 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14463	13057	0	17	1364	25

⚪ 2024-09-02 09:16:41 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-09-02 09:23:54 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
51 (only retried tests)	33	0	2	1	15

⚪ 2024-09-02 09:24:02 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-09-02 09:32:19 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
24 (only retried tests)	8	0	2	0	14

🟢 2024-09-02 09:32:27 UTC Build successful.
🟢 2024-09-02 09:33:07 UTC ydbd size 8.3 GiB changed* by +38.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `4cbf358`	merge: `24b8935`	diff	diff %
ydbd size	8 907 349 176 Bytes	8 907 388 664 Bytes	+38.6 KiB	+0.000%
ydbd stripped size	481 310 568 Bytes	481 314 568 Bytes	+3.9 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-02T08:13:15Z

⚪ 2024-09-02 08:13:14 UTC Pre-commit check linux-x86_64-release-asan for 24b8935 has started.
⚪ 2024-09-02 08:16:04 UTC ya make is running...
🔴 2024-09-02 09:54:39 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10056	9969	0	36	25	26

🟢 2024-09-02 09:55:34 UTC Build successful.
🟢 2024-09-02 09:56:06 UTC ydbd size 5.5 GiB changed* by +46.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `4cbf358`	merge: `24b8935`	diff	diff %
ydbd size	5 947 599 336 Bytes	5 947 646 672 Bytes	+46.2 KiB	+0.001%
ydbd stripped size	1 488 307 984 Bytes	1 488 320 464 Bytes	+12.2 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

status `WriteId` is ignored

github-actions · 2024-09-02T10:34:42Z

⚪ 2024-09-02 10:34:41 UTC Pre-commit check linux-x86_64-release-clang14 for 67753e3 has started.
⚪ 2024-09-02 10:37:34 UTC ya make is running...
🟢 2024-09-02 10:44:47 UTC Build successful.

github-actions · 2024-09-02T10:39:53Z

⚪ 2024-09-02 10:39:53 UTC Pre-commit check linux-x86_64-release-asan for 67753e3 has started.
⚪ 2024-09-02 10:42:40 UTC ya make is running...
⚫ 2024-09-02 11:20:36 UTC Check cancelled

github-actions · 2024-09-02T10:58:11Z

⚪ 2024-09-02 10:58:10 UTC Pre-commit check linux-x86_64-relwithdebinfo for 67753e3 has started.
⚪ 2024-09-02 11:00:56 UTC ya make is running...
⚫ 2024-09-02 11:20:51 UTC Check cancelled

github-actions · 2024-09-02T11:22:49Z

⚪ 2024-09-02 11:22:48 UTC Pre-commit check linux-x86_64-release-asan for 66b7f48 has started.
⚪ 2024-09-02 11:26:35 UTC ya make is running...
🔴 2024-09-02 12:50:15 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10156	10094	0	19	25	18

🟢 2024-09-02 12:51:10 UTC Build successful.
🟢 2024-09-02 12:51:40 UTC ydbd size 5.5 GiB changed* by +39.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `ebfe3cd`	merge: `66b7f48`	diff	diff %
ydbd size	5 953 849 840 Bytes	5 953 890 392 Bytes	+39.6 KiB	+0.001%
ydbd stripped size	1 489 337 360 Bytes	1 489 344 656 Bytes	+7.1 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-02T11:29:47Z

⚪ 2024-09-02 11:29:47 UTC Pre-commit check linux-x86_64-relwithdebinfo for 66b7f48 has started.
⚪ 2024-09-02 11:32:33 UTC ya make is running...
🟡 2024-09-02 12:36:17 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
39189	33755	0	13	5401	20

⚪ 2024-09-02 12:39:40 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-09-02 12:47:14 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
32 (only retried tests)	17	0	0	0	15

🟢 2024-09-02 12:47:21 UTC Build successful.
🟢 2024-09-02 12:47:56 UTC ydbd size 8.3 GiB changed* by +54.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `ebfe3cd`	merge: `66b7f48`	diff	diff %
ydbd size	8 916 405 512 Bytes	8 916 461 024 Bytes	+54.2 KiB	+0.001%
ydbd stripped size	481 603 144 Bytes	481 606 984 Bytes	+3.8 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-02T11:29:49Z

⚪ 2024-09-02 11:29:49 UTC Pre-commit check linux-x86_64-release-clang14 for 66b7f48 has started.
⚪ 2024-09-02 11:32:32 UTC ya make is running...
🟢 2024-09-02 11:38:43 UTC Build successful.

…tform#8517)

…8670)

…tform#8517) (ydb-platform#8670)

Alek5andr-Kotov requested a review from a team as a code owner August 30, 2024 09:54

Alek5andr-Kotov requested a review from nshestakov August 30, 2024 09:54

github-actions bot added the bugfix label Aug 30, 2024

Alek5andr-Kotov commented Aug 30, 2024

View reviewed changes

Alek5andr-Kotov added 5 commits September 2, 2024 13:29

[-] if WriteId is deleted then TEvProposeTransactionResult(ABORTED)

9ca047e

[*] if the message TEvProposeTransaction is received, the subscription

fea30c1

status `WriteId` is ignored

[*] WriteIdIsDisabled

625af60

[*] refactoring tests

853e486

[+] the WriteId field in the TEvProposeTransaction message

3332d6e

Alek5andr-Kotov force-pushed the LOGBROKER-9232 branch from 619fd7a to 3332d6e Compare September 2, 2024 10:29

Alek5andr-Kotov changed the title ~~WIP: the race between 'TEvProposeTransaction and 'TEvLockStatus~~ The race between 'TEvProposeTransaction and TEvLockStatus` Sep 2, 2024

github-actions bot removed the bugfix label Sep 2, 2024

Alek5andr-Kotov changed the title ~~The race between 'TEvProposeTransaction and TEvLockStatus`~~ The race between TEvProposeTransaction and TEvLockStatus Sep 2, 2024

github-actions bot added bugfix and removed bugfix labels Sep 2, 2024

[*] function has been renamed

ef9e251

github-actions bot added bugfix and removed bugfix labels Sep 2, 2024

alexnick88 approved these changes Sep 2, 2024

View reviewed changes

Alek5andr-Kotov merged commit 03395eb into ydb-platform:main Sep 2, 2024
11 of 13 checks passed

Alek5andr-Kotov added a commit to Alek5andr-Kotov/ydb that referenced this pull request Sep 3, 2024

The race between TEvProposeTransaction and TEvLockStatus (ydb-pla…

ceaa5d0

…tform#8517)

KirillKurdyukov pushed a commit to KirillKurdyukov/ydb that referenced this pull request Sep 4, 2024

The race between TEvProposeTransaction and TEvLockStatus (ydb-pla…

0089c81

…tform#8517)

siarheivesialou mentioned this pull request Sep 4, 2024

fix memory leak in ymq proxy #8706

Merged

Alek5andr-Kotov added a commit that referenced this pull request Sep 4, 2024

The race between TEvProposeTransaction and TEvLockStatus (#8517) (#…

ae19a36

…8670)

This was referenced Sep 4, 2024

YQ-3151 added issues for timeout and cancelled #8718

Merged

YQ kqprun added cancel after for script requests #8768

Merged

Fixed unit test ManyColumnShards #8777

Merged

niksaveliev mentioned this pull request Sep 9, 2024

Fix CLI max partitions default #8944

Merged

spuchin pushed a commit to spuchin/ydb that referenced this pull request Sep 9, 2024

The race between TEvProposeTransaction and TEvLockStatus (ydb-pla…

a4817e0

…tform#8517) (ydb-platform#8670)

niksaveliev mentioned this pull request Sep 12, 2024

Fix reading without consumer from federation #9136

Merged

StekPerepolnen mentioned this pull request Sep 17, 2024

doc hc api images source #9376

Merged

This was referenced Oct 29, 2024

viewer healthcheck pathmerge stable 24 3 11 hotfix #11042

Closed

viewer healthcheck pathmerge stable 24 #11044

Closed

serbel324 mentioned this pull request Nov 19, 2024

Merge double accelerations into 24-3 #11740

Merged

GrigoriyPA mentioned this pull request Feb 21, 2025

merge to ydb stable YQ-4070 supported insert values for s3 #14914

Closed

The race between TEvProposeTransaction and TEvLockStatus #8517

The race between TEvProposeTransaction and TEvLockStatus #8517

Uh oh!

Conversation

Alek5andr-Kotov commented Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog entry

Changelog category

Additional information

Uh oh!

github-actions bot commented Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Alek5andr-Kotov commented Aug 30, 2024

Uh oh!

Alek5andr-Kotov Aug 30, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

The race between `TEvProposeTransaction` and `TEvLockStatus` #8517

The race between `TEvProposeTransaction` and `TEvLockStatus` #8517

Alek5andr-Kotov commented Aug 30, 2024 •

edited

Loading

github-actions bot commented Aug 30, 2024 •

edited

Loading

github-actions bot commented Aug 30, 2024 •

edited

Loading

github-actions bot commented Aug 30, 2024 •

edited

Loading

github-actions bot commented Aug 30, 2024 •

edited

Loading

github-actions bot commented Aug 30, 2024 •

edited

Loading

github-actions bot commented Aug 30, 2024 •

edited

Loading

github-actions bot commented Sep 2, 2024 •

edited

Loading

github-actions bot commented Sep 2, 2024 •

edited

Loading

github-actions bot commented Sep 2, 2024 •

edited

Loading

github-actions bot commented Sep 2, 2024 •

edited

Loading

github-actions bot commented Sep 2, 2024 •

edited

Loading

github-actions bot commented Sep 2, 2024 •

edited

Loading

github-actions bot commented Sep 2, 2024 •

edited

Loading

github-actions bot commented Sep 2, 2024 •

edited

Loading

github-actions bot commented Sep 2, 2024 •

edited

Loading