New kafka metering fix #7856
Closed
New kafka metering fix #7856
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
niksaveliev
commented
Aug 15, 2024
niksaveliev
commented
- Only members of ReleaseApprovers team can approve changes in this branch (Only members of ReleaseApprovers team can approve changes in this branch #6323)
- Initial commit for stable-24-3 (Initial commit for stable-24-3 #6321)
- fix query errors processing (fix query errors processing #6441) (fix query errors processing (#6441) #6451)
- Merge Handle empty GC commands correctly #6398 Fix VDisk replication stuck sensor bug #6405 from main into stable-24-3 (Merge #6398 #6405 from main into stable-24-3 #6448)
- Fix test_db_counters test (Fix test_db_counters test #6436)
- 24-3 Fix tests with EnableLocalDBBtreeIndex = false (24-3 Fix tests with EnableLocalDBBtreeIndex = false #6391)
- 24-3 Iterative B-Tree histograms builder (24-3 Iterative B-Tree histograms builder #6392)
- 24-3 Obtain UsedTableMemory from private cache stats, include historic indexes size (24-3 Obtain UsedTableMemory from private cache stats, include historic indexes size #6400)
- revert 19bed6c (revert 19bed6c19f373cc26a3b8d1d4e4d1ff1126a56ce #6469)
- 24-3: Fix bugs in coordinator state migration (24-3: Fix bugs in coordinator state migration #6460)
- Merge some fixes (Merge some fixes #6490)
- set predicate-extract config default (set predicate-extract config default #6456)
- 24-3: Fix bugs in: change exchange split, removing schema snapshots & emitting of resolved timestamps (24-3: Fix bugs in: change exchange split, removing schema snapshots & emitting of resolved timestamps #6616)
- 24.3: Fix null dereference in node broker (KIKIMR-21693) (KIKIMR-21693: Fix null dereference in node broker #6516) (24.3: Fix null dereference in node broker (KIKIMR-21693) (#6516) #6623)
- Use the whole TTableDescription to describe an indexImplTable in TIndexDescription (Use the whole TTableDescription to describe an indexImplTable in TIndexDescription #6280) (Use the whole TTableDescription to describe an indexImplTable in TIndexDescription (#6280) #6450)
- Changed memory calculation for TEvFreeItems (merge from main Changed memory calculation for TEvFreeItems #6625) (Changed memory calculation for TEvFreeItems (merge from main #6625) #6694)
- fix drain compatability (fix drain compatability #6613) (fix drain compatability (#6613) #6716)
- fix neighbours count (fix neighbours count #6409) (fix neighbours count (#6409) #6718)
- fix TabletsTotal counter (fix TabletsTotal counter #6492) (fix TabletsTotal counter (#6492) #6717)
- do not trigger emergency balancer when all nodes have high usage (do not trigger emergency balancer when all nodes have high usage #6532) (do not trigger emergency balancer when all nodes have high usage (#6532) #6720)
- 24-3: schemeshard: preserialize Table.SplitBoundary for describe result (24-3: schemeshard: preserialize Table.SplitBoundary for describe result #6847)
- 24-3: schemeshard: reject operations with too big local tx commit (schemeshard: reject operations with too big local tx commit #6760) (24-3: schemeshard: reject operations with too big local tx commit (#6760) #6849)
- 24-3: Fix read requirements on init (24-3: Fix read requirements on init #6948)
- build: refresh Embedded UI (v6.11.0) (build: refresh Embedded UI (v6.11.0) #6968)
- Extend DisabledOnSchemeShard FF for column tables on dedicated bases … (Extend DisabledOnSchemeShard FF for column tables on dedicated bases … #6962)
- Merge Fix LWTrace leaking HTTP request string to HTML page (Fix LWTrace leaking HTTP request string to HTML page #7011) (Fix LWTrace leaking HTTP request string to HTML page (#7011) #7014) (Merge Fix LWTrace leaking HTTP request string to HTML page (#7011) (#7014) #7024)
- deprioritise system tablets in balancer (deprioritise system tablets in balancer #6840) (deprioritise system tablets in balancer (#6840) #7047)
- use potential max thread count in local (use potential max thread count in local #7050)
- merge memory limitations features and staff to stable-24-3 (merge memory limitations features and staff to stable-24-3 #6803)
- Fix scope id handling (Fix scope id handling #7093)
- YQ-3345 support load cpu threshold (YQ-3345 support load cpu threshold #6790)
- YQ WM fixed build errors (YQ WM fixed build errors #7128)
- Disable modification in cs by default (Disable modification in cs by default #7161)
- CS improvements (CS improvements #7013)
- Dont create delete flags column if not necessary (Dont create delete flags column if not necessary #7169)
- 24-3: Enable/disable ssl connections, return connection_string in API (24-3: Enable/disable ssl connections, return connection_string in API #7147)
- 24-3: Check shard state at TTxCdcStreamEmitHeartbeats (24-3: Check shard state at TTxCdcStreamEmitHeartbeats #7148)
- 24-3: Allow streams on index tables, replicate index tables (24-3: Allow streams on index tables, replicate index tables #7150)
- add autopartitioning by load (add autopartitioning by load #7124)
- fix viewer redirects (fix viewer redirects #7103) (fix viewer redirects (#7103) #7120)
- 24-3 Freeze event spaces ids (24-3 Freeze event spaces ids #7192)
- Added metadata flag to track if stats for optimizer were loaded (Added metadata flag to track if stats for optimizer were loaded #7190)
- fix compaction intervals construction (fix compaction intervals construction #7176) (fix compaction intervals construction (#7176) #7208)
- add enable implicit params flag to table service config (add enable implicit params flag to table service config #6943)
- 24-3 Fix using uninitialized value error. (24-3 Fix using uninitialized value error. #7224)
- 24-3 [CBO] Warning FIX ([CBO] Warning FIX #6420) (24-3 [CBO] Warning FIX (#6420) #7241)
- fix segfault and memory leaks in CS (fix segfault and memory leaks in CS #7288)
- 24-3 Rescue asan tests (remove NO_EXPORT_DYNAMIC_SYMBOLS for them) (24-3 Rescue asan tests (remove NO_EXPORT_DYNAMIC_SYMBOLS for them) #7311)
- Enable sequences (Enable sequences #7297)
- YDB-7262 Add default permille ICB configuration for vdisk garbage compaction threshold (YDB-7262 Add default permille ICB configuration for vdisk garbage compaction threshold #7319)
- WM fixed bugs and performance (WM fixed bugs and performance #7254)
- hotfix for hotkeys (hotfix for hotkeys #7425)
- Fix actor system usage after it was freed (Fix actor system usage after it was freed #7316) (Fix actor system usage after it was freed (#7316) #7404)
- Disable vpatch 24-3 (Disable vpatch 24-3 #7320)
- PushLeftStage fix (PushLeftStage fix #7349)
- Ignore stored GCBarrierPreparation (Ignore stored GCBarrierPreparation #7473)
- add config option to specify TTL for user logins (add config option to specify TTL for user logins #7083) (add config option to specify TTL for user logins (#7083) #7486)
- fix for dead nodes always passing filter (fix for dead nodes always passing filter #6912) (fix for dead nodes always passing filter (#6912) #7522)
- stable-24-3: Enable COUNT in view queries (Enable COUNT in view queries #6820) (stable-24-3: Enable COUNT in view queries (#6820) #7523)
- Fix trim duration (change microseconds to milliseconds) (Fix trim duration (change microseconds to milliseconds) #7302)
- 24-3: Sinks improvements (24-3: Sinks improvements #6856)
- 24-3: CTAS fixes (24-3: CTAS fixes #6857)
- 24-3: Datashard + Columnshard Reads (24-3: Datashard + Columnshard Reads #6858)
- storage healthcheck fixes (fix missing static group issue #7212) (storage healthcheck fixes (#7212) #7394)
- control inflight pings in hive (control inflight pings in hive #6916) (control inflight pings in hive (#6916) #7238)
- Delete empty portions normalizer (Delete empty portions normalizer #7596) (Delete empty portions normalizer (#7596) #7600)
- fix suspended read session of a topic (fix suspended read session of a topic #7635)
- Allow logging AST with free arguments. (Allow logging AST with free arguments. #7573) (Allow logging AST with free arguments. (#7573) #7653)
- YDB-2568 Enable match_recognize in ydb / stable-24-3 (YDB-2568 Enable match_recognize in ydb / stable-24-3 #7540)
- 24-3: Fix unexpected read iterator stream reset (24-3: Fix unexpected read iterator stream reset #7710)
- Allow multiple join-broadcasts in single stage (Allow multiple join-broadcasts in single stage #7556) (Allow multiple join-broadcasts in single stage (#7556) #7713)
- [Http] Reply with structured issues when client accepts json data ([Http] Reply with structured issues when client accepts json data #7691)
- fix LockedPartitions without session (fix LockedPartitions without session #7736)
- fix query id for recompilation (fix query id for recompilation #7599)
- [CDC] Do not lose presition during float/double to json serialization… ([CDC] Do not lose presition during float/double to json serialization… #7737)
- Fix json float/double print format (Fix json float/double print format #7572) (Fix json float/double print format (#7572) #7776)
- Double free when accessing .AsBoxed() on Pod with 0 refcount (Double free when accessing .AsBoxed() on Pod with 0 refcount #7659) (Double free when accessing .AsBoxed() on Pod with 0 refcount (#7659) #7721)
- Fix RU for kafka read
Co-authored-by: Tony-Romanov <[email protected]>
… emitting of resolved timestamps (ydb-platform#6616)
…form#6625) (ydb-platform#6694) Co-authored-by: mregrock <[email protected]> Co-authored-by: Egor Kulin <[email protected]>
…lt (ydb-platform#6847) merged 83a86c2 from main Preserialize table's split boundaries the same way table partitions are. The size of both depend on the same variable: number of shards in the table, but TablePartitions was preserialized (and cached) while Table.SplitBoundaries wasn't. Preserializing all potentially huge parts of DescribeSchemeResult message before it gets to the interconnect saves interconnect actors additional serialization costs. And when partitioning of the huge tables goes through the period of a rapid change that additional serialization causes interconnect to overload. Single shortcoming though: preserialized SplitBoundary is not used (cannot be used) when boundaries of the index tables are requested through describe request on table index. KIKIMR-21686
…b-platform#6760) (ydb-platform#6849) merged ebee36a from main Add commit redo size check for successfully ignited operations as a precaution measure to avoid infinite loop of schemeshard hitting local tx commit redo size limit, restarting, attempting to propose persisted operation again, hitting commit redo size limit again, restarting and so on. This could happen with inherently massive operations such as copy-tables used as a starting step of database export/backup. Coping large number of tables with huge number of partitions can result in so large TTxOperationPropose local transaction that its size would hit the limit imposed by the tablet executor. Tablet violating that limit is considered broken and will be immediately stopped. See ydb/core/tablet_flat/flat_executor.cpp, NTabletFlatExecutor::TExecutor::ExecuteTransaction(). KIKIMR-21751
Co-authored-by: antonkovalenko <[email protected]>
Co-authored-by: kruall <[email protected]>
…#7302) Co-authored-by: Vlad Kuznecov <[email protected]>
Co-authored-by: uzhastik <[email protected]>
…orm#7653) Co-authored-by: Tony-Romanov <[email protected]>
…tform#7659) (ydb-platform#7721) Co-authored-by: Александр Новожилов <[email protected]>
|
🔴 Unable to merge your PR into the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.