Initial commit

Author: anton-bobkov

ydb/docs/en/core/dev/index.md

@@ -27,4 +27,6 @@ Main resources:
   - [{#T}](../postgresql/intro.md)
   - [{#T}](../reference/kafka-api/index.md)
 
+- [{#T}](troubleshooting/index.md)
+
 If you're interested in developing {{ ydb-short-name }} core or satellite projects, refer to the [documentation for contributors](../contributor/index.md).

ydb/docs/en/core/dev/toc_p.yaml

@@ -18,6 +18,11 @@ items:
     path: primary-key/toc_p.yaml
 - name: Secondary indexes
   href: secondary-indexes.md
+- name: Troubleshooting
+  href: troubleshooting/index.md
+  include:
+    mode: link
+    path: troubleshooting/toc_p.yaml
 - name: Query plans optimization
   href: query-plans-optimization.md
 - name: Batch upload

ydb/docs/en/core/dev/troubleshooting/index.md

@@ -0,0 +1,5 @@
+# Troubleshooting
+
+This section of {{ ydb-short-name }} documentation covers everything you need to know to troubleshoot {{ ydb-short-name }} issues.
+
+- [{#T}](performance/index.md)

ydb/docs/en/core/dev/troubleshooting/performance/hardware/_assets/cpu-read-only-tx-latency.png

@@ -0,0 +1,24 @@
+1. Analyze CPU utilization in all pools:
+
+    1. Open the **CPU** dashboard in Grafana.
+
+    1. See if the following charts show any spikes:
+
+        - **CPU by execution pool** chart
+        - **User pool - CPU by host** chart
+        - **System pool - CPU by host** chart
+        - **Batch pool - CPU by host** chart
+        - **IC pool - CPU by host** chart
+        - **IO pool - CPU by host** chart
+
+1. Analyze changes in the user load that might have caused the CPU bottleneck. See the following charts on the **DB overview** in Grafana:
+
+    - **Requests** chart
+
+    - **Request size** chart
+
+    - **Response size** chart
+
+    Also, see all of the charts in the **Operations** section of the **DataShard** dashboard. These charts show the number of rows processed per query.
+
+1. Contact your DBA and inquire about {{ ydb-short-name }} backups.

ydb/docs/en/core/dev/troubleshooting/performance/hardware/_assets/cpu-row-read-rows.png

@@ -0,0 +1,18 @@
+1. Open the **DB overview** dashboard in Grafana.
+
+1. On the **Cost and DiskTimeAvailable relation** chart, see if the **Disk cost** spikes cross the **DiskTimeAvailable** level.
+
+    ![](../_assets/disk-time-available--disk-cost.png)
+
+    This chart shows the estimated total bandwith capacity of the storage system in conventional units (green) and the average usage cost (blue). When the average usage cost exceeds the total bandwidth capacity, the storage system of {{ ydb-short-name }} gets overloaded, which results in higher latencies.
+
+1. On the **Burst duration, ms** chart, check for any spikes of the load on the storage system. This chart shows microbursts of the load on the storage system, in microseconds.
+
+    ![](../_assets/microbursts.png)
+
+    {% note info %}
+
+    This chart might show microbursts of the load that are not detected by the average usage cost in the **Cost and DiskTimeAvailable relation** chart.
+
+    {% endnote %}
+

ydb/docs/en/core/dev/troubleshooting/performance/hardware/_assets/disk-time-available--disk-cost.png

@@ -0,0 +1,25 @@
+# CPU bottleneck
+
+High CPU usage can lead to slow query processing and increased response times. When CPU resources are constrained, the database may have difficulty handling complex queries or large transaction volumes.
+
+The CPU resources are mainly used by the actor system. Depending on the type, all actors run in one of the following pools:
+
+- **System**: A pool that is designed for running quick internal operations in YDB (it serves system tablets, state storage, distributed storage I/O, and erasure coding).
+
+- **User**: A pool that serves the user load (user tablets, queries run in the Query Processor).
+Batch: A pool that serves tasks with no strict limit on the execution time, background operations like garbage collection and heavy queries run in the Query Processor.
+
+- **IO**: A pool responsible for performing any tasks with blocking operations (such as authentication or writing logs to a file).
+
+- **IC**: Interconnect, it serves the load related to internode communication (system calls to wait for sending and send data across the network, data serialization, as well as message splits and merges).
+
+## Diagnostics
+
+<!-- The include is added to allow partial overrides in overlays  -->
+{% include notitle [#](_includes/cpu-bottleneck.md) %}
+
+<!-- If the spikes on these charts align, the increased latencies may be related to the higher number of rows being read from the database. In this case, the available database nodes might not be sufficient to handle the increased load. -->
+
+## Recommendation
+
+Add additional [database nodes](../../../../concepts/glossary.md#database-node) or allocate more CPU cores to the existing nodes.

ydb/docs/en/core/dev/troubleshooting/performance/hardware/_assets/microbursts.png

@@ -0,0 +1,31 @@
+# Disk space
+
+A lack of available disk space can prevent the database from storing new data, resulting in the database becoming read-only. This can also cause slowdowns as the system tries to reclaim disk space by compacting existing data more aggressively.
+
+## Diagnostics
+
+<!-- TODO: Mention the limits metric, if it's operational -->
+
+1. See if the **DB overview > Storage** charts in Grafana show any spikes.
+
+1. In [Embedded UI](../../../../reference/embedded-ui/index.md), on the **Storage** tab, analyze the list of available storage groups and nodes and their disk usage.
+
+    {% note tip %}
+
+    Use the **Out of Space** filter to list only the storage groups with full disks.
+
+    {% endnote %}
+
+    ![](_assets/storage-groups-disk-space.png)
+
+{% note info %}
+
+You can also use the [Healthcheck API](../../../../reference/ydb-sdk/health-check-api.md) in your application to get this information.
+
+{% endnote %}
+
+## Recommendations
+
+Add more [storage groups](../../../../concepts/glossary.md#storage-group) to the database.
+
+If the cluster doesn't have spare storage groups, configure them first. Add additional [storage nodes](../../../../concepts/glossary.md#storage-node), if necessary.

ydb/docs/en/core/dev/troubleshooting/performance/hardware/_assets/storage-groups-disk-space.png

@@ -0,0 +1,37 @@
+# Insufficient memory (RAM)
+
+If [swap](https://en.wikipedia.org/wiki/Memory_paging#Unix_and_Unix-like_systems) (paging of anonymous memory) is disabled on the server running {{ ydb-short-name }}, insufficient memory activates another kernel feature called the [OOM killer](https://en.wikipedia.org/wiki/Out_of_memory), which terminates the most memory-intensive processes (often the database itself). This feature also interacts with [cgroups](https://en.wikipedia.org/wiki/Cgroups) if multiple cgroups are configured.
+
+If swap is enabled, insufficient memory may cause the database to rely heavily on disk I/O, which is significantly slower than accessing data directly from memory. This can result in increased latencies during query execution and data retrieval.
+
+Additionally, which components within the  {{ ydb-short-name }} process consume memory may also be significant.
+
+## Diagnostics
+
+1. Determine whether any {{ ydb-short-name }} nodes recently restarted for unknown reasons. Exclude cases of {{ ydb-short-name }} upgrades.
+
+    1. Open [Embedded UI](../../../../reference/embedded-ui/index.md).
+
+    1. On the **Nodes** tab, look for nodes that have low uptime.
+
+1. Determine whether memory usage reached 100%.
+
+    1. Open the **DB overview** dashboard in Grafana.
+
+    1. Analyze the charts in the **Memory** section.
+
+1. Determine whether the user load on {{ ydb-short-name }} has increased. Analyze the following charts on the **DB overview** dashboard in Grafana:
+
+    - **Requests** chart
+    - **Request size** chart
+    - **Response size** chart
+
+1. Determine whether new releases or data usage changes occurred in your applications.
+
+## Recommendation
+
+Consider the following solutions to the problem of insufficient memory:
+
+- If the load on {{ ydb-short-name }} increased because of new usage patterns or higher volume of queries, try to optimize the application to reduce the load on {{ ydb-short-name }} or add more {{ ydb-short-name }} nodes.
+
+- If the load on {{ ydb-short-name }} has not changed, but {{ ydb-short-name }} nodes still restart, consider adding more {{ ydb-short-name }} nodes or raising the hard memory limit available to {{ ydb-short-name }} nodes.

ydb/docs/en/core/dev/troubleshooting/performance/hardware/_includes/cpu-bottleneck.md

@@ -0,0 +1,15 @@
+# I/O bandwidth
+
+A high rate of read/write operations can overwhelm the disk subsystem, resulting in increased latencies in data access. When the system cannot read or write data quickly enough, queries that depend on disk access will experience delays.
+
+## Diagnostics
+
+<!-- The include is added to allow partial overrides in overlays  -->
+{% include notitle [io-bandwidth](./_includes/io-bandwidth.md) %}
+
+## Recommendations
+
+Add more [storage groups](../../../../concepts/glossary.md#storage-group) to the database.
+
+In case of high microburst values, you can also try to balance the load across storage groups.
+

ydb/docs/en/core/dev/troubleshooting/performance/hardware/_includes/io-bandwidth.md

@@ -0,0 +1,87 @@
+# Troubleshooting performance issues
+
+Addressing database performance issues often requires a holistic approach, which includes optimizing queries, properly configuring hardware resources, and ensuring that both the database and the application are well-designed. Regular monitoring and maintenance are essential for proactively identifying and resolving these issues.
+
+## Tools to troubleshoot performance issues
+
+Troubleshooting performance issues in {{ ydb-short-name }} involves the following tools:
+
+- [{{ ydb-short-name }} metrics](../../../reference/observability/metrics/index.md)
+
+- [{{ ydb-short-name }} logs](../../../devops/manual/logging.md)
+
+- [{{ ydb-short-name }} CLI](../../../reference/ydb-cli/index.md)
+
+- [Tracing](../../../reference/observability/tracing/setup.md)
+
+- [Embedded UI](../../../reference/embedded-ui/index.md)
+
+- [Query plans](../../query-plans-optimization.md)
+
+
+## Classification of {{ ydb-short-name }} performance issues
+
+Database performance issues can be classified into several categories based on their nature. This documentation section provides a high-level overview of these categories, starting with the lowest layers of the system and going all the way to the client. Below is a separate section for the [actual performance troubleshooting instructions](#instructions).
+
+- **Hardware infrastructure issues**.
+
+    - **[Network issues](infrastructure/network.md)**. Insufficient bandwidth or network congestion in data centers can significantly affect {{ ydb-short-name }} performance.
+
+    - **[Data center outages](infrastructure/dc-outage.md)**: Disruptions in data center operations that can cause service or data unavailability. These outages may result from various factors, such as power failures, natural disasters, or cyber-attacks. A common fault-tolerant setup for {{ ydb-short-name }} spans three data centers or availability zones (AZs). {{ ydb-short-name }} can continue operating without interruption, even if one data center and a server rack in another are lost. However, it will initiate the relocation of tablets from the offline AZ to the remaining online nodes, temporarily leading to higher query latencies. Distributed transactions involving tablets that are moving to other nodes might experience increased latencies.
+
+    - **[Data center maintenance and drills](infrastructure/dc-drills.md)**. Planned maintenance or drills, exercises conducted to prepare personnel for potential emergencies or outages, can also affect query performance. Depending on the maintenance scope or drill scenario, some {{ ydb-short-name }} servers might become unavailable, which leads to the same impact as an outage.
+
+    - **[Server hardware issues](infrastructure/hardware.md)**. Malfunctioning CPU, memory modules, and network cards, until replaced, significantly impact database performance up to the total unavailability of the affected server.
+
+- **Insufficient resources**. These issues refer to situations when the workload demands more physical resources — such as CPU, memory, disk space, and network bandwidth — than allocated to a database.
+
+    - **[CPU bottlenecks](hardware/cpu-bottleneck.md)**. High CPU usage can result in slow query processing and increased response times. When CPU resources are limited, the database may struggle to handle complex queries or large transaction loads.
+
+    - **[Insufficient disk space](hardware/disk-space.md)**. A lack of available disk space can prevent the database from storing new data, resulting in the database becoming read-only. This can also cause slowdowns as the system tries to reclaim disk space by compacting existing data more aggressively.
+
+    - **[Insufficient memory (RAM)](hardware/insufficient-memory.md)**. If swap is disabled, insufficient memory can trigger [OOM killer](https://en.wikipedia.org/wiki/Out_of_memory) that terminates the most memory-hungry processes (for servers running databases, it's often the database itself). If swap is enabled, insufficient memory can cause the database to rely heavily on disk I/O for operations, which is significantly slower than accessing data from memory. This can lead to increased latencies in query execution and data retrieval.
+
+    - **[Insufficient disk I/O bandwidth](hardware/io-bandwidth.md)**. High read/write operations can overwhelm disk subsystems, leading to increased latencies in data access. When the system cannot read or write data quickly enough, queries that require disk access will be delayed.
+
+- **OS issues**
+
+    - **Hardware resource allocation issues**. Suboptimal allocation of resources, for example poorly configured control groups (cgroups), may result in insufficient resources for {{ ydb-short-name }} and increase query latencies even though physical hardware resources are still available on the database server.
+
+    - **[System clock drift](system/system-clock-drift.md)**. If the system clocks on the {{ ydb-short-name }} servers start to drift apart, it will lead to increased distributed transaction latencies. In severe cases, {{ ydb-short-name }} might even refuse to process distributed transactions and return errors.
+
+    - Other processes running on the same nodes as YDB, such as antiviruses, observability agents, etc.
+
+    - Kernel misconfiguration.
+
+- **YDB-related issues**
+
+    - **[Rolling restart](system/ydb-updates.md)**. {{ ydb-short-name }} is a distributed system that supports rolling restart, when database administrators update {{ ydb-short-name }} nodes one by one. This helps keep the {{ ydb-short-name }} cluster up and running during the update process or some {{ ydb-short-name }} configuration changes. However, when a YDB node is being restarted, Hive moves the tables that run on this node to other nodes, and that may lead to increased latencies for queries that are processed by the moving tables.
+
+    - Actor system pools misconfiguration.
+
+    - SDK usage issues (maybe worth being a separate category).
+
+- **Schema design issues**. These issues stem from inefficient decisions made during the creation of tables and indices. They can significantly impact query performance.
+
+- **Client application issues**. These issues refer to database queries executing slower than expected because of their inefficient design.
+
+## Instructions {#instructions}
+
+To troubleshoot {{ ydb-short-name }} performance issues, treat each potential cause as a hypothesis. Systematically review the list of hypotheses and verify whether they apply to your situation. The documentation for each cause provides a description, guidance on how to check diagnostics, and recommendations on what to do if the hypothesis is confirmed.
+
+If any known changes occurred in the system around the time the performance issues first appeared, investigate those first. Otherwise, follow this recommended order for evaluating potential root causes. This order is loosely based on the descending frequency of their occurrence on large production {{ ydb-short-name }} clusters.
+
+1. [Overloaded shards](schemas/overloaded-shards.md)
+1. [Excessive tablet splits and merges](schemas/splits-merges.md)
+1. [Frequent tablet moves between nodes](system/tablets-moved.md)
+1. Insufficient hardware resources:
+    - [Disk I/O bandwidth](hardware/io-bandwidth.md)
+    - [Disk space](hardware/disk-space.md)
+    - [Insufficient CPU](hardware/cpu-bottleneck.md)
+1. [Hardware issues](infrastructure/hardware.md) and [data center outages](infrastructure/dc-outage.md)
+1. [Network issues](infrastructure/network.md)
+1. [{{ ydb-short-name }} updates](system/ydb-updates.md)
+1. [System clock drift](system/system-clock-drift.md)
+
+
+

ydb/docs/en/core/dev/troubleshooting/performance/hardware/cpu-bottleneck.md

@@ -0,0 +1,9 @@
+To determine if one of the data centers of the {{ ydb-short-name }} cluster is not available, follow these steps:
+
+1. Open [Embedded UI](../../../../../reference/embedded-ui/index.md).
+
+1. On the **Nodes** tab, analyze the [health indicators](../../../../../reference/embedded-ui/ydb-monitoring.md#colored_indicator) in the **Host** and **DC** columns.
+
+    ![](../_assets/cluster-nodes.png)
+
+    If all of the nodes in one of the DCs (data centers) are not available, this data center is most likely offline.

ydb/docs/en/core/dev/troubleshooting/performance/hardware/disk-space.md

@@ -0,0 +1,11 @@
+To diagnose network issues, use the healthcheck in the [Embedded UI](../../../../../reference/embedded-ui/index.md):
+
+1. In the [Embedded UI](../../../../../reference/embedded-ui/index.md), go to the **Databases** tab and click on the database.
+
+1. On the **Navigation** tab, ensure the required database is selected.
+
+1. Open the **Diagnostics** tab.
+
+1. On the **Network** tab, select the **With problems** filter.
+
+    ![](../_assets/diagnostics-network.png)

ydb/docs/en/core/dev/troubleshooting/performance/hardware/insufficient-memory.md

@@ -0,0 +1,11 @@
+# Data center maintenance and drills
+
+Planned maintenance or drills, exercises conducted to prepare personnel for potential emergencies or outages, can also affect query performance. Depending on the maintenance scope or drill scenario, some {{ ydb-short-name }} nodes might become unavailable, which leads to the same impact as an [outage](./dc-outage.md).
+
+## Diagnostics
+
+Check the planned maintenance and drills schedules to see if their timelines match with observed performance issues, otherwise, check the [datacenter outage recommendations](dc-outage.md).
+
+## Recommendations
+
+Contact the person responsible for the current maintenance or drill to discuss whether the performance impact is severe enough for it to be finished/canceled early, if possible.

ydb/docs/en/core/dev/troubleshooting/performance/hardware/io-bandwidth.md

@@ -0,0 +1,14 @@
+# Data center outages
+
+Data center outages are disruptions in data center operations that could cause service or data unavailability, but {{ ydb-short-name }} has means to avoid it. Various factors, such as power failures, natural disasters, or cyberattacks, may cause these outages. A common fault-tolerant setup for {{ ydb-short-name }} spans three data centers or availability zones (AZs). {{ ydb-short-name }} can maintain uninterrupted operation even if one data center and a server rack in another are lost. However, it will initiate the relocation of tablets from the offline AZ to the remaining online nodes, temporarily leading to higher query latencies. Distributed transactions involving tablets that are moving to other nodes might experience increased latencies.
+
+## Diagnostics
+
+<!-- The include is added to allow partial overrides in overlays  -->
+{% include notitle [dc-outage](_includes/dc-outage.md) %}
+
+## Recommendations
+
+Contact the responsible party for the affected data center to resolve the underlying issue. If you are part of a larger organization, this could be an in-house team managing low-level infrastructure. Otherwise, contact the cloud service or hosting provider's support service.
+
+Meanwhile, check the data center's status page if it has one.