Skip to content

Add --backport-to-template functionality #15483

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 11, 2025

Conversation

Jorres
Copy link
Collaborator

@Jorres Jorres commented Mar 8, 2025

Changelog entry

  • added option --backport-to-template to ydb_configure

Changelog category

  • New feature

Description for reviewers

  • --backport-to-template: new boolean flag that will attempt to add generated blob_storage_config into cluster_config_template.yaml so you don't have to do it by hand. It will do it without excessive diff, template formatting will be kept, only the blob_storage_config section will be changed
  • added support for bootstrap_config, now this can be specified in config.yaml directly:
bootstrap_config:
  shared_cache_config:
    memory_limit: 100000000000

Earlier, template.yaml format was different, just like for many other options:

shared_cache:
  memory_limit: 200000000000

  • it is now possible to specify blob_storage_config without a static group, and ydb_configure will correctly keep those settings while generating a static group. For example:
blob_storage_config:
  vdisk_performance_settings:
    vdisk_types:
    - min_huge_blob_size_in_bytes: 100000
      pdisk_type: ROT
    - min_huge_blob_size_in_bytes: 100000
      pdisk_type: UNKNOWN_TYPE

will turn into

blob_storage_config:
  service_set: # generated service_set
    availability_domains:
    - '1'
    groups:
    - erasure_species: 4
      group_generation: 1
      group_id: 0
      rings:
      - fail_domains:
        - vdisk_locations:
          - node_id: 1
            path: /dev/disk/by-partlabel/ydb_disk_ssd_01_01
            pdisk_category: 1
            pdisk_guid: 8882996296504668847
            pdisk_id: 1
...
        - vdisk_locations:
          - node_id: 8
            path: /dev/disk/by-partlabel/ydb_disk_ssd_01_01
            pdisk_category: 1
            pdisk_guid: 17495542895019359475
            pdisk_id: 1
  vdisk_performance_settings: # kept this from template
    vdisk_types:
    - min_huge_blob_size_in_bytes: 100000
      pdisk_type: ROT
    - min_huge_blob_size_in_bytes: 100000
      pdisk_type: UNKNOWN_TYPE

Copy link

github-actions bot commented Mar 8, 2025

🟢 2025-03-10 12:31:18 UTC The validation of the Pull Request description is successful.

Copy link

github-actions bot commented Mar 8, 2025

2025-03-08 19:30:43 UTC Pre-commit check linux-x86_64-release-asan for f2c3988 has started.
2025-03-08 19:30:58 UTC Artifacts will be uploaded here
2025-03-08 19:33:20 UTC ya make is running...
🟡 2025-03-08 19:57:39 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
1748 1710 0 16 11 11

2025-03-08 19:57:56 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-03-08 20:09:23 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
44 (only retried tests) 37 0 0 0 7

🟢 2025-03-08 20:09:30 UTC Build successful.
🟢 2025-03-08 20:09:46 UTC ydbd size 3.7 GiB changed* by -96 Bytes, which is <= 0 Bytes vs main: OK

ydbd size dash main: cc84759 merge: f2c3988 diff diff %
ydbd size 3 996 545 000 Bytes 3 996 544 904 Bytes -96 Bytes -0.000%
ydbd stripped size 1 389 195 176 Bytes 1 389 195 112 Bytes -64 Bytes -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Mar 8, 2025

2025-03-08 19:32:52 UTC Pre-commit check linux-x86_64-relwithdebinfo for f2c3988 has started.
2025-03-08 19:33:06 UTC Artifacts will be uploaded here
2025-03-08 19:35:23 UTC ya make is running...
🟡 2025-03-08 20:01:19 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
4823 4705 0 1 36 81

2025-03-08 20:01:48 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-03-08 20:05:52 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
95 (only retried tests) 21 0 0 0 74

🟢 2025-03-08 20:05:59 UTC Build successful.
🟢 2025-03-08 20:06:10 UTC ydbd size 2.1 GiB changed* by 0 Bytes, which is <= 0 Bytes vs main: OK

ydbd size dash main: cc84759 merge: f2c3988 diff diff %
ydbd size 2 294 860 176 Bytes 2 294 860 176 Bytes 0 Bytes 0.000%
ydbd stripped size 480 611 392 Bytes 480 611 392 Bytes 0 Bytes 0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Mar 10, 2025

2025-03-10 12:29:55 UTC Pre-commit check linux-x86_64-relwithdebinfo for e78d008 has started.
2025-03-10 12:30:02 UTC Artifacts will be uploaded here
2025-03-10 12:32:27 UTC ya make is running...
🟢 2025-03-10 12:58:49 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
4823 4704 0 0 36 83

🟢 2025-03-10 12:59:26 UTC Build successful.
🟢 2025-03-10 12:59:39 UTC ydbd size 2.1 GiB changed* by 0 Bytes, which is <= 0 Bytes vs main: OK

ydbd size dash main: 2ce497a merge: e78d008 diff diff %
ydbd size 2 293 856 560 Bytes 2 293 856 560 Bytes 0 Bytes 0.000%
ydbd stripped size 480 432 384 Bytes 480 432 384 Bytes 0 Bytes 0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Mar 10, 2025

2025-03-10 12:30:44 UTC Pre-commit check linux-x86_64-release-asan for e78d008 has started.
2025-03-10 12:30:51 UTC Artifacts will be uploaded here
2025-03-10 12:33:07 UTC ya make is running...
🟡 2025-03-10 12:59:21 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
1748 1729 0 9 3 7

2025-03-10 12:59:39 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-03-10 13:11:16 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
22 (only retried tests) 16 0 0 0 6

🟢 2025-03-10 13:11:22 UTC Build successful.
🟢 2025-03-10 13:11:38 UTC ydbd size 3.7 GiB changed* by -96 Bytes, which is <= 0 Bytes vs main: OK

ydbd size dash main: 2ce497a merge: e78d008 diff diff %
ydbd size 3 994 884 584 Bytes 3 994 884 488 Bytes -96 Bytes -0.000%
ydbd stripped size 1 388 591 688 Bytes 1 388 591 624 Bytes -64 Bytes -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@Jorres Jorres requested review from shmel1k, bma13 and sourcecd March 10, 2025 14:15
@Jorres Jorres merged commit c805655 into ydb-platform:main Mar 11, 2025
14 checks passed
adameat pushed a commit to adameat/ydb that referenced this pull request Mar 20, 2025
lberserq pushed a commit to lberserq/ydb that referenced this pull request Mar 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants