Skip to content

ydbd_slice: add support for using raw config.yaml #15094

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Feb 27, 2025
Merged

ydbd_slice: add support for using raw config.yaml #15094

merged 3 commits into from
Feb 27, 2025

Conversation

melazyk
Copy link
Collaborator

@melazyk melazyk commented Feb 26, 2025 •
edited
Loading

ydbd_slice: add support for using raw config.yaml via --yaml-config and dynconfig.yaml via --yaml-dynconfig

Changelog entry

  • Add yaml_configurator instead of configurator
  • Add warning for old style cluster.yaml
  • Add dynconfig-generator for generating simple dynconfig.yaml
  • Add --save-raw-cfg option for saving raw config from old cluster.yaml

Changelog category

  • Experimental feature

Description for reviewers

This PR introduces a new slice with the following changes:

  1. Configuration Files:

    • Created config.yaml and cluster.yaml (containing only the domains section).
    • Generated the minimally required dynconfig.yaml using: 
      ydbd_slice dynconfig-generator --yaml-config config.yaml --output-file dynconfig.yaml

  2. Slice Creation:

    • The slice is created using the following command: 
      ydbd_slice install cluster.yaml all --yaml-config config.yaml --binary /path/to/ydbd

  3. Notes:

    • If ydbd_slice is run without the --yaml-config parameter, a warning with instructions will be displayed.

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 26, 2025 •
edited
Loading

🟢 2025-02-27 15:12:32 UTC The validation of the Pull Request description is successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 26, 2025 •
edited
Loading

⚪ 2025-02-26 16:48:03 UTC Pre-commit check linux-x86_64-release-asan for 6c97ed1 has started.
⚪ 2025-02-26 16:48:17 UTC Artifacts will be uploaded here
⚪ 2025-02-26 16:50:38 UTC ya make is running...
🟡 2025-02-26 17:15:06 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
1751 1715 0 13 16 7

⚪ 2025-02-26 17:15:31 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-02-26 17:26:58 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
41 (only retried tests) 33 0 3 1 4

⚪ 2025-02-26 17:27:06 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-02-26 17:31:38 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
12 (only retried tests) 7 0 2 0 3

🟢 2025-02-26 17:31:45 UTC Build successful.
🟡 2025-02-26 17:32:00 UTC ydbd size 3.7 GiB changed* by +158.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: e0f8b20 merge: 6c97ed1 diff diff %
ydbd size 3 982 265 248 Bytes 3 982 427 488 Bytes +158.4 KiB +0.004%
ydbd stripped size 1 387 038 472 Bytes 1 387 092 456 Bytes +52.7 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 26, 2025 •
edited
Loading

⚪ 2025-02-26 16:48:14 UTC Pre-commit check linux-x86_64-relwithdebinfo for 6c97ed1 has started.
⚪ 2025-02-26 16:48:28 UTC Artifacts will be uploaded here
⚪ 2025-02-26 16:50:53 UTC ya make is running...
🟡 2025-02-26 17:15:57 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
4818 4699 0 1 36 82

⚪ 2025-02-26 17:16:50 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-02-26 17:19:58 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
94 (only retried tests) 18 0 1 0 75

⚪ 2025-02-26 17:20:08 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-02-26 17:23:02 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
88 (only retried tests) 13 0 1 0 74

🟢 2025-02-26 17:23:10 UTC Build successful.
🟢 2025-02-26 17:23:22 UTC ydbd size 2.1 GiB changed* by +75.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: e0f8b20 merge: 6c97ed1 diff diff %
ydbd size 2 284 785 120 Bytes 2 284 862 488 Bytes +75.6 KiB +0.003%
ydbd stripped size 479 196 384 Bytes 479 214 240 Bytes +17.4 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

kobzonega
kobzonega reviewed Feb 27, 2025
View reviewed changes
@melazyk melazyk force-pushed the ydbd_slice-configurator-remove branch from f58c5eb to e24f1a1 Compare February 27, 2025 13:33
kobzonega
kobzonega previously approved these changes Feb 27, 2025
View reviewed changes
@melazyk melazyk enabled auto-merge (squash) February 27, 2025 13:46
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 27, 2025 •
edited
Loading

⚪ 2025-02-27 14:46:58 UTC Pre-commit check linux-x86_64-release-asan for 5e7869d has started.
⚪ 2025-02-27 14:47:14 UTC Artifacts will be uploaded here
⚪ 2025-02-27 14:49:36 UTC ya make is running...
🟡 2025-02-27 15:15:22 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
1751 1735 0 5 3 8

⚪ 2025-02-27 15:15:45 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-02-27 15:27:02 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
20 (only retried tests) 12 0 2 1 5

⚪ 2025-02-27 15:27:10 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-02-27 15:39:57 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
12 (only retried tests) 7 0 1 0 4

🟢 2025-02-27 15:40:06 UTC Build successful.
🟢 2025-02-27 15:40:22 UTC ydbd size 3.7 GiB changed* by -128 Bytes, which is <= 0 Bytes vs main: OK

ydbd size dash main: c5b2980 merge: 5e7869d diff diff %
ydbd size 3 985 411 216 Bytes 3 985 411 088 Bytes -128 Bytes -0.000%
ydbd stripped size 1 387 765 576 Bytes 1 387 765 512 Bytes -64 Bytes -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 27, 2025 •
edited
Loading

⚪ 2025-02-27 14:46:59 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5e7869d has started.
⚪ 2025-02-27 14:47:13 UTC Artifacts will be uploaded here
⚪ 2025-02-27 14:49:38 UTC ya make is running...
🟡 2025-02-27 15:15:00 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
4818 4687 0 1 47 83

⚪ 2025-02-27 15:15:36 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-02-27 15:18:11 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
107 (only retried tests) 29 0 1 0 77

⚪ 2025-02-27 15:18:20 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-02-27 15:21:27 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
92 (only retried tests) 16 0 1 0 75

🟢 2025-02-27 15:21:35 UTC Build successful.
🟢 2025-02-27 15:21:47 UTC ydbd size 2.1 GiB changed* by -64 Bytes, which is <= 0 Bytes vs main: OK

ydbd size dash main: c5b2980 merge: 5e7869d diff diff %
ydbd size 2 286 685 160 Bytes 2 286 685 096 Bytes -64 Bytes -0.000%
ydbd stripped size 479 417 824 Bytes 479 417 824 Bytes 0 Bytes 0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@melazyk melazyk disabled auto-merge February 27, 2025 15:50
@melazyk
ydbd_slice: add support for using raw config.yaml via --yaml-config a…
8644423 
…nd dynconfig.yaml via --yaml-dynconfig

- Add yaml_configurator instead of configurator
- Add warning for old style cluster.yaml
- Add dynconfig-generator for generating simple dynconfig.yaml
- Add --save-raw-cfg option for saving raw config from old cluster.yaml
@melazyk
ydbd_slice: review fixes
091339a 
- --yaml-dynconfig removed
- dynconfig.yaml generator fixed ( remove storage only options )
- fix the typo
@melazyk
ydbd_slice: yaml_configurator.py flake8 test fix
eb7ce73
@melazyk melazyk force-pushed the ydbd_slice-configurator-remove branch from e24f1a1 to eb7ce73 Compare February 27, 2025 16:21
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 27, 2025 •
edited
Loading

⚪ 2025-02-27 16:23:24 UTC Pre-commit check linux-x86_64-relwithdebinfo for a2d04ce has started.
⚪ 2025-02-27 16:24:03 UTC Artifacts will be uploaded here
⚪ 2025-02-27 16:26:32 UTC ya make is running...
🟡 2025-02-27 16:50:58 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
4818 4697 0 1 36 84

⚪ 2025-02-27 16:51:40 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-02-27 17:03:36 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
98 (only retried tests) 20 0 0 0 78

🟢 2025-02-27 17:03:45 UTC Build successful.
🟢 2025-02-27 17:03:57 UTC ydbd size 2.1 GiB changed* by 0 Bytes, which is <= 0 Bytes vs main: OK

ydbd size dash main: 6abead2 merge: a2d04ce diff diff %
ydbd size 2 286 685 096 Bytes 2 286 685 096 Bytes 0 Bytes 0.000%
ydbd stripped size 479 417 824 Bytes 479 417 824 Bytes 0 Bytes 0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 27, 2025 •
edited
Loading

⚪ 2025-02-27 16:23:46 UTC Pre-commit check linux-x86_64-release-asan for a2d04ce has started.
⚪ 2025-02-27 16:24:01 UTC Artifacts will be uploaded here
⚪ 2025-02-27 16:26:25 UTC ya make is running...
🟡 2025-02-27 16:51:17 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
1751 1724 0 7 11 9

⚪ 2025-02-27 16:51:36 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-02-27 17:02:52 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
29 (only retried tests) 23 0 1 0 5

⚪ 2025-02-27 17:03:04 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-02-27 17:07:47 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10 (only retried tests) 6 0 0 0 4

🟢 2025-02-27 17:07:53 UTC Build successful.
🟢 2025-02-27 17:08:13 UTC ydbd size 3.7 GiB changed* by -96 Bytes, which is <= 0 Bytes vs main: OK

ydbd size dash main: 6abead2 merge: a2d04ce diff diff %
ydbd size 3 985 411 184 Bytes 3 985 411 088 Bytes -96 Bytes -0.000%
ydbd stripped size 1 387 765 576 Bytes 1 387 765 512 Bytes -64 Bytes -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@melazyk melazyk enabled auto-merge (squash) February 27, 2025 17:08
@melazyk melazyk merged commit d052910 into ydb-platform:main Feb 27, 2025
12 checks passed
@melazyk melazyk deleted the ydbd_slice-configurator-remove branch February 27, 2025 17:12
lberserq pushed a commit to lberserq/ydb that referenced this pull request Mar 3, 2025
@melazyk
ydbd_slice: add support for using raw config.yaml (ydb-platform#15094)
5af5627
@GrigoriyPA GrigoriyPA mentioned this pull request Mar 4, 2025
Closed
This was referenced Mar 10, 2025
Merged
Merged
@alexnick88 alexnick88 mentioned this pull request Mar 19, 2025
Closed
blinkov pushed a commit that referenced this pull request Mar 21, 2025
@melazyk @blinkov
ydbd_slice: add support for using raw config.yaml (#15094)
5c9b62a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kobzonega kobzonega kobzonega approved these changes

Assignees
No one assigned
Labels
experimental-feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@melazyk @kobzonega