feat(data_integrity_trails): add logging of acquired and broken locks #13164

ulya-sidorina · 2025-01-03T18:58:55Z

No description provided.

ydb/core/tx/datashard/datashard_integrity_trails.h

ydb/core/tx/datashard/datashard_kqp.cpp

ydb/core/kqp/ut/data_integrity/kqp_data_integrity_trails_ut.cpp

github-actions · 2025-02-28T11:13:58Z

⚪ 2025-02-28 11:13:57 UTC Pre-commit check linux-x86_64-release-asan for 95f8276 has started.
⚪ 2025-02-28 11:14:13 UTC Artifacts will be uploaded here
⚪ 2025-02-28 11:17:04 UTC ya make is running...
🟡 2025-02-28 12:31:23 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11729	11491	0	169	31	38

⚪ 2025-02-28 12:32:35 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-02-28 12:47:56 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
310 (only retried tests)	259	0	11	7	33

⚪ 2025-02-28 12:48:06 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-02-28 13:00:31 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
87 (only retried tests)	54	0	1	1	31

🟢 2025-02-28 13:00:38 UTC Build successful.
🟡 2025-02-28 13:01:10 UTC ydbd size 3.7 GiB changed* by +195.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `0db71ba`	merge: `95f8276`	diff	diff %
ydbd size	3 986 703 248 Bytes	3 986 903 400 Bytes	+195.5 KiB	+0.005%
ydbd stripped size	1 388 116 936 Bytes	1 388 170 312 Bytes	+52.1 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-02-28T11:13:58Z

⚪ 2025-02-28 11:13:58 UTC Pre-commit check linux-x86_64-relwithdebinfo for 95f8276 has started.
⚪ 2025-02-28 11:14:12 UTC Artifacts will be uploaded here
⚪ 2025-02-28 11:17:04 UTC ya make is running...
🟡 2025-02-28 12:06:56 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
19204	17673	0	2	1455	74

⚪ 2025-02-28 12:08:36 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-02-28 12:19:39 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
389 (only retried tests)	255	0	2	7	125

⚪ 2025-02-28 12:19:52 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-02-28 12:31:06 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
190 (only retried tests)	64	0	2	0	124

🟢 2025-02-28 12:31:14 UTC Build successful.
🟡 2025-02-28 12:31:40 UTC ydbd size 2.1 GiB changed* by +116.7 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `0db71ba`	merge: `95f8276`	diff	diff %
ydbd size	2 287 412 760 Bytes	2 287 532 304 Bytes	+116.7 KiB	+0.005%
ydbd stripped size	479 533 664 Bytes	479 553 568 Bytes	+19.4 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

snaury · 2025-03-03T07:36:15Z

Я аппрувнул, но там один из новых тестов не проходит, возможно с изменением логики логгирования что-то поменялось, или может быть там есть какой-то баг. Кстати я не уверен покрыты ли все требуемые места, т.к. локи могут ломаться например bulk upsert'ами и erase'ами, там тоже есть ApplyLocks, но логирование видимо забыто? Или то что ломается не транзакциями не интересно?

github-actions · 2025-03-03T11:05:28Z

⚪ 2025-03-03 11:05:27 UTC Pre-commit check linux-x86_64-relwithdebinfo for 585ee12 has started.
⚪ 2025-03-03 11:05:58 UTC Artifacts will be uploaded here
⚪ 2025-03-03 11:10:28 UTC ya make is running...
🟡 2025-03-03 12:01:55 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
19165	17812	0	2	1216	135

⚪ 2025-03-03 12:03:49 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-03-03 12:16:41 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
202 (only retried tests)	75	0	0	0	127

🟢 2025-03-03 12:16:52 UTC Build successful.
🟡 2025-03-03 12:17:11 UTC ydbd size 2.1 GiB changed* by +116.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `1f60fc8`	merge: `585ee12`	diff	diff %
ydbd size	2 288 460 760 Bytes	2 288 580 408 Bytes	+116.8 KiB	+0.005%
ydbd stripped size	479 649 216 Bytes	479 669 120 Bytes	+19.4 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-03-03T11:06:51Z

⚪ 2025-03-03 11:06:50 UTC Pre-commit check linux-x86_64-release-asan for 585ee12 has started.
⚪ 2025-03-03 11:07:04 UTC Artifacts will be uploaded here
⚪ 2025-03-03 11:09:50 UTC ya make is running...
🟡 2025-03-03 12:17:23 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11686	11520	0	103	25	38

⚪ 2025-03-03 12:18:25 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-03-03 12:32:37 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
224 (only retried tests)	178	0	8	7	31

⚪ 2025-03-03 12:32:48 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-03-03 12:44:44 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
75 (only retried tests)	40	0	0	5	30

🟢 2025-03-03 12:44:54 UTC Build successful.
🟡 2025-03-03 12:45:21 UTC ydbd size 3.7 GiB changed* by +193.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `6afe3c8`	merge: `585ee12`	diff	diff %
ydbd size	3 988 492 304 Bytes	3 988 690 488 Bytes	+193.5 KiB	+0.005%
ydbd stripped size	1 388 505 480 Bytes	1 388 555 976 Bytes	+49.3 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

ulya-sidorina · 2025-03-03T12:51:15Z

Я аппрувнул, но там один из новых тестов не проходит, возможно с изменением логики логгирования что-то поменялось, или может быть там есть какой-то баг. Кстати я не уверен покрыты ли все требуемые места, т.к. локи могут ломаться например bulk upsert'ами и erase'ами, там тоже есть ApplyLocks, но логирование видимо забыто? Или то что ломается не транзакциями не интересно?

Упал старый тест KqpDataIntegrityTrails::UpsertEvWriteQueryService+isOlap как раз из-за изменения логики логирования, поправила

На данный момент нас интересуют только локи, сломанные в транзакции (т к механизм data integrity логов поддержан пока только для транзакционных запросов)

…#13164)

…ydb-platform#13164)

ulya-sidorina force-pushed the feature/add_locks_to_logs branch 2 times, most recently from d62cdb4 to 7d7c53e Compare January 6, 2025 18:24

ulya-sidorina changed the title ~~feat(data_integrity_trails): add lock logging~~ feat(data_integrity_trails): add logging of acquired locks Jan 6, 2025

ulya-sidorina force-pushed the feature/add_locks_to_logs branch from 7d7c53e to d3c4465 Compare January 6, 2025 18:27

ulya-sidorina marked this pull request as ready for review January 7, 2025 11:01

ulya-sidorina force-pushed the feature/add_locks_to_logs branch 3 times, most recently from e23df93 to 9ee9382 Compare January 15, 2025 12:42

ulya-sidorina force-pushed the feature/add_locks_to_logs branch from 9ee9382 to 5a16ec0 Compare February 18, 2025 11:28

ulya-sidorina requested a review from a team as a code owner February 18, 2025 11:28

ulya-sidorina force-pushed the feature/add_locks_to_logs branch 14 times, most recently from d7856be to b85bce5 Compare February 24, 2025 12:06

ulya-sidorina changed the title ~~feat(data_integrity_trails): add logging of acquired locks~~ feat(data_integrity_trails): add logging of acquired and broken locks Feb 24, 2025

ulya-sidorina requested a review from azevaykin February 24, 2025 12:14

snaury reviewed Feb 25, 2025

View reviewed changes

ydb/core/tx/datashard/datashard_integrity_trails.h Show resolved Hide resolved

snaury reviewed Feb 25, 2025

View reviewed changes

ydb/core/tx/datashard/datashard_kqp.cpp Outdated Show resolved Hide resolved

snaury reviewed Feb 25, 2025

View reviewed changes

ydb/core/tx/datashard/datashard_kqp.cpp Outdated Show resolved Hide resolved

azevaykin reviewed Feb 25, 2025

View reviewed changes

ydb/core/kqp/ut/data_integrity/kqp_data_integrity_trails_ut.cpp Show resolved Hide resolved

ydb-platform deleted a comment from github-actions bot Feb 28, 2025

ulya-sidorina force-pushed the feature/add_locks_to_logs branch from 0b2654a to 123fa59 Compare February 28, 2025 10:48

ulya-sidorina requested review from snaury and azevaykin February 28, 2025 10:49

ulya-sidorina force-pushed the feature/add_locks_to_logs branch from 123fa59 to ddcb619 Compare February 28, 2025 11:12

ydb-platform deleted a comment from github-actions bot Feb 28, 2025

snaury previously approved these changes Mar 3, 2025

View reviewed changes

feat(data_integrity_trails): add logging of acquired locks

d595583

ulya-sidorina dismissed snaury’s stale review via d595583 March 3, 2025 11:04

ulya-sidorina force-pushed the feature/add_locks_to_logs branch from ddcb619 to d595583 Compare March 3, 2025 11:04

ulya-sidorina requested review from snaury and azevaykin and removed request for azevaykin March 3, 2025 12:42

snaury approved these changes Mar 3, 2025

View reviewed changes

va-kuznecov approved these changes Mar 4, 2025

View reviewed changes

ulya-sidorina merged commit 9eea4e3 into ydb-platform:main Mar 4, 2025
12 checks passed

GrigoriyPA mentioned this pull request Mar 4, 2025

YQ-3655 added block splitting into DQ output channel #15330

Closed

This was referenced Mar 10, 2025

Fix segfault in Group Layout Sanitizer, add stress tests with INACTIVE/FAULTY disks #15516

Merged

Add compaction selected strategy counters #15689

Merged

alexnick88 mentioned this pull request Mar 19, 2025

provide database in schemecache request #15938

Closed

blinkov pushed a commit that referenced this pull request Mar 21, 2025

feat(data_integrity_trails): add logging of acquired and broken locks (…

bc6af69

…#13164)

dorooleg pushed a commit to dorooleg/ydb that referenced this pull request Apr 19, 2025

feat(data_integrity_trails): add logging of acquired and broken locks (…

bdab698

…ydb-platform#13164)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(data_integrity_trails): add logging of acquired and broken locks #13164

feat(data_integrity_trails): add logging of acquired and broken locks #13164

Uh oh!

ulya-sidorina commented Jan 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Feb 28, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Feb 28, 2025 •

edited

Loading

Uh oh!

snaury commented Mar 3, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Mar 3, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Mar 3, 2025 •

edited

Loading

Uh oh!

ulya-sidorina commented Mar 3, 2025

Uh oh!

Uh oh!

Uh oh!

feat(data_integrity_trails): add logging of acquired and broken locks #13164

feat(data_integrity_trails): add logging of acquired and broken locks #13164

Uh oh!

Conversation

ulya-sidorina commented Jan 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

snaury commented Mar 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ulya-sidorina commented Mar 3, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Feb 28, 2025 •

edited

Loading

github-actions bot commented Feb 28, 2025 •

edited

Loading

snaury commented Mar 3, 2025 •

edited

Loading

github-actions bot commented Mar 3, 2025 •

edited

Loading

github-actions bot commented Mar 3, 2025 •

edited

Loading