YugabyteDB Performance Monitoring: Kill Slow SQL in 6 Steps

At 02:14 the p99 alert fires. Dashboards show the cluster is up, CPU looks unremarkable and every node is healthy. Yet checkout queries that took 8 ms now take 400. In a single-node database you would open one tool and see the problem. In a distributed SQL database the slow part could be on any of nine nodes, in any of thousands of tablets, inside a Raft round trip, a lock queue or a disk read. YugabyteDB performance monitoring is the discipline of finding which one, quickly and with evidence.

This guide is the YugabyteDB observability playbook we use for that hunt. It maps every telemetry surface YugabyteDB exposes to the layer it describes, shows how Active Session History correlates a YSQL query with the DocDB RPCs it caused, and ends with a six-step troubleshooting workflow and a read-only evidence collector you can run during an incident. All facts come from the official YugabyteDB documentation and are version-pinned; numbers in examples are illustrative, not benchmarks.

Why YugabyteDB observability needs a map

YugabyteDB is a PostgreSQL-compatible query layer (YSQL) running on DocDB, where every table and index is split into tablets, each tablet is a Raft group, and each replica stores data in a RocksDB-based LSM tree. A single SELECT can touch several tablets on several nodes. So Yugabyte performance troubleshooting is never "look at the slow log": you need to know which layer was waiting, then which node, then which tablet.

The good news is that YugabyteDB instruments every layer. The challenge is that the instruments live in different places: SQL views, Prometheus endpoints on three ports, web UIs and diagnostic bundles. Figure 1 is the map we start from in every YugabyteDB performance monitoring engagement.

Layer 1: YugabyteDB performance monitoring with metrics and golden signals

According to the YugabyteDB metrics documentation, the servers export close to 2,000 metrics in JSON (/metrics) and Prometheus (/prometheus-metrics) formats: YB-Master on port 7000, YB-TServer on 9000, YSQL on 13000 and YCQL on 12000. Names follow <category>_<server_type>_<service_type>_<method>, so handler_latency_yb_ysqlserver_SQLProcessor_SelectStmt is the latency histogram of YSQL SELECT processing, and handler_latency_yb_tserver_TabletServerService_Read is the storage-layer read RPC handler.

Before building dashboards, check what each endpoint actually returns. On clusters with many tablets the tserver endpoint can be large, which is why v2.20.2 added filter parameters such as version=v2, table_blocklist and max_metric_entries.

#!/usr/bin/env bash
# YugabyteDB performance monitoring: probe every metrics endpoint on one node.
# Read-only. Host names come from the environment; nothing is hardcoded.
set -euo pipefail
: "${YB_NODE:?set YB_NODE to a node address}"

# Ports (defaults): yb-master UI 7000, yb-tserver UI 9000,
# YSQL metrics 13000, YCQL metrics 12000.
for port in 7000 9000 13000; do
  echo "== ${YB_NODE}:${port} =="
  # Prometheus exposition format; count series to size your scrape budget.
  curl -s "http://${YB_NODE}:${port}/prometheus-metrics" | grep -vc '^#'
done

# v2.20.2+: filter the tserver scrape. version=v2 enables the allow/block lists,
# so table-level series can be dropped on clusters with thousands of tablets.
curl -s "http://${YB_NODE}:9000/prometheus-metrics?version=v2&table_blocklist=.*&max_metric_entries=5000" \
  | grep -E '^(handler_latency_yb_tserver_TabletServerService_(Read|Write)|rocksdb_block_cache_(hit|miss)|log_sync_latency)' \
  | head -20

# YSQL statement latency and connection gauges live on the YSQL metrics port.
curl -s "http://${YB_NODE}:13000/prometheus-metrics" \
  | grep -E '^(handler_latency_yb_ysqlserver_SQLProcessor_(SelectStmt|InsertStmt|UpdateStmt)|yb_ysqlserver_(active_connection|connection|max_connection)_total)'

A Prometheus configuration for YugabyteDB performance monitoring then needs one job per server type, each with metrics_path: /prometheus-metrics:

# prometheus.yml - YugabyteDB performance monitoring scrape jobs.
# One job per server type keeps label sets clean and lets you scrape
# the heavy tserver endpoint less often if you need to.
scrape_configs:
  - job_name: yb-master
    metrics_path: /prometheus-metrics
    static_configs:
      - targets: ['yb-node-1:7000', 'yb-node-2:7000', 'yb-node-3:7000']

  - job_name: yb-tserver
    metrics_path: /prometheus-metrics
    params:
      version: ['v2']
      table_blocklist: ['.*']      # drop per-table series; keep server-level
    scrape_interval: 30s
    static_configs:
      - targets: ['yb-node-1:9000', 'yb-node-2:9000', 'yb-node-3:9000']

  - job_name: yb-ysql
    metrics_path: /prometheus-metrics
    static_configs:
      - targets: ['yb-node-1:13000', 'yb-node-2:13000', 'yb-node-3:13000']

With the data flowing, six queries cover the golden signals: throughput and latency at the YSQL layer, read RPC rate at the storage layer, block cache efficiency, catalog cache misses and connection pressure. The throughput and latency metrics page documents the per-statement handlers, the cache and storage metrics page the RocksDB and WAL series, and the connection metrics page the YSQL connection gauges.

# YugabyteDB performance monitoring: the golden-signal PromQL set.
# Metric names as documented by Yugabyte; latency values are in microseconds.

# 1. YSQL throughput: SELECTs per second, per node
sum by (instance) (rate(handler_latency_yb_ysqlserver_SQLProcessor_SelectStmt_total_count[1m]))

# 2. YSQL average SELECT latency (us), per node
sum by (instance) (rate(handler_latency_yb_ysqlserver_SQLProcessor_SelectStmt_total_sum[1m]))
  /
sum by (instance) (rate(handler_latency_yb_ysqlserver_SQLProcessor_SelectStmt_total_count[1m]))

# 3. Storage-layer read RPCs per second (DocDB IOPS seen by the tserver)
sum by (instance) (rate(handler_latency_yb_tserver_TabletServerService_Read_total_count[1m]))

# 4. Block cache hit ratio (falls -> RocksDB_ReadBlockFromFile rises in ASH)
sum by (instance) (rate(rocksdb_block_cache_hit[5m]))
  /
(sum by (instance) (rate(rocksdb_block_cache_hit[5m])) + sum by (instance) (rate(rocksdb_block_cache_miss[5m])))

# 5. Catalog cache misses per second (rises with connection churn and DDL)
sum by (instance) (rate(handler_latency_yb_ysqlserver_SQLProcessor_CatalogCacheTableMisses_count[5m]))

# 6. Connection pressure: active vs configured maximum, and rejections
sum by (instance) (yb_ysqlserver_active_connection_total)
  / sum by (instance) (yb_ysqlserver_max_connection_total)
increase(yb_ysqlserver_connection_over_limit_total[10m])

In YugabyteDB performance monitoring, metrics answer "is something wrong, and where" well. They do not answer "why". A rising average SELECT latency on one node tells you where to look; it does not tell you whether the time went to a lock, a disk read or a remote Raft quorum. That is the job of the next two layers. Alerts should therefore be few and tied to a runbook step:

# yb-alerts.yml - YugabyteDB performance monitoring alert rules.
# Thresholds are illustrative starting points; derive yours from a baseline.
groups:
  - name: yugabytedb-performance
    rules:
      - alert: YSQLSelectLatencyHigh
        expr: |
          sum by (instance) (rate(handler_latency_yb_ysqlserver_SQLProcessor_SelectStmt_total_sum[5m]))
          / sum by (instance) (rate(handler_latency_yb_ysqlserver_SQLProcessor_SelectStmt_total_count[5m]))
          > 20000                       # 20 ms average, in microseconds
        for: 10m
        labels: {severity: warning}
        annotations:
          summary: "YSQL SELECT latency high on {{ $labels.instance }}"
          runbook: "Rank statements with pg_stat_statements, then group ASH by wait_event."

      - alert: BlockCacheHitRatioLow
        expr: |
          sum by (instance) (rate(rocksdb_block_cache_hit[10m]))
          / (sum by (instance) (rate(rocksdb_block_cache_hit[10m]))
             + sum by (instance) (rate(rocksdb_block_cache_miss[10m]))) < 0.90
        for: 15m
        labels: {severity: warning}

      - alert: YSQLConnectionsRejected
        expr: increase(yb_ysqlserver_connection_over_limit_total[5m]) > 0
        labels: {severity: critical}
        annotations:
          summary: "Connections rejected at ysql_max_connections on {{ $labels.instance }}"

Two more series deserve a panel. log_sync_latency and log_append_latency show how long the Raft log takes to append and fsync, which bounds write latency on every tablet leader. The UpdateConsensus handler latency measures leader-to-follower replication; the Raft and distributed system metrics page also warns that hybrid clock skew above 500 ms may compromise consistency guarantees, so skew belongs on the dashboard too.

Layer 2: statement-level YugabyteDB observability with pg_stat_statements

YugabyteDB extends the familiar pg_stat_statements view with columns that only make sense in a distributed database: yb_latency_histogram, a JSONB histogram of execution latencies read with yb_get_percentile(), and DocDB RPC counters such as docdb_read_rpcs, docdb_write_rpcs, docdb_rows_scanned, docdb_rows_returned and docdb_wait_time. The RPC statistics are governed by yb_enable_pg_stat_statements_rpc_stats, on by default from v2026.1.1.0.

Averages hide tail latency, and tail latency is what users feel. The histogram lets YugabyteDB performance monitoring rank statements by p99 rather than mean, and the DocDB columns explain whether the time was spent in YSQL or waiting on storage.

-- YugabyteDB performance monitoring: rank statements by where the time went.
-- The docdb_* RPC columns are collected when yb_enable_pg_stat_statements_rpc_stats
-- is on (default true from v2026.1.1.0); yb_latency_histogram is YugabyteDB-specific.
SELECT queryid,
       calls,
       round(total_exec_time::NUMERIC, 1)                         AS total_ms,
       round((total_exec_time / NULLIF(calls, 0))::NUMERIC, 2)    AS mean_ms,
       yb_get_percentile(yb_latency_histogram, 99)                AS p99_ms,
       round(docdb_read_rpcs::NUMERIC  / NULLIF(calls, 0), 1)     AS read_rpcs_per_call,
       round(docdb_rows_scanned::NUMERIC / NULLIF(docdb_rows_returned, 0), 1)
                                                                  AS scanned_per_returned,
       round(docdb_wait_time::NUMERIC, 1)                         AS docdb_wait_ms,
       left(query, 70)                                            AS query
FROM pg_stat_statements
WHERE calls > 50
ORDER BY total_exec_time DESC
LIMIT 15;

-- Reading the result:
--   high p99_ms but normal mean_ms    -> tail problem: contention, hot tablet, GC of a node
--   read_rpcs_per_call grows with rows -> per-row RPCs: missing index or no batched join
--   scanned_per_returned >> 1         -> filter applied after the scan: fix the index
--   docdb_wait_ms close to total_ms    -> time is in storage/network, not in YSQL CPU

-- Isolate an incident window (shared memory is per node; reset on each node):
-- SELECT pg_stat_statements_reset();

Two YugabyteDB observability cautions. pg_stat_statements is per node, so a statement executed through a load balancer has its statistics spread across every YSQL server; collect from all nodes before you conclude. And the default pg_stat_statements.max of 5000 entries can be exhausted by applications that do not use bind parameters, which is itself a finding worth reporting.

Layer 3: Yugabyte performance troubleshooting with Active Session History

Active Session History is the most important YugabyteDB observability feature for troubleshooting. It samples sessions that are on CPU or waiting in an RPC, across YSQL backends, YCQL and YB-TServer RPC threads, and stores the samples in a per-node circular buffer exposed as yb_active_session_history. Per the ASH monitoring guide, ysql_yb_enable_ash is on by default, sampling runs every 1,000 ms (ysql_yb_ash_sampling_interval_ms) with up to 500 events per interval (ysql_yb_ash_sample_size), and the buffer is sized by ysql_yb_ash_circular_buffer_size.

What makes ASH powerful in a distributed system is correlation. Each sample carries a root_request_id shared by the YSQL backend and every tserver RPC it triggered, a query_id that matches pg_stat_statements.queryid, a top_level_node_id that joins to yb_servers(), and for TServer events a wait_event_aux holding the first 15 characters of the tablet ID.

Start Yugabyte performance troubleshooting with load over time, grouped by wait class. A sudden band of lock or consensus waits on top of a steady CPU baseline is usually the incident. Sum sample_weight rather than counting rows, because a sample can stand for more than one session when sampling is capped.

-- YugabyteDB performance monitoring: "database load" by wait class per minute.
-- Each ASH sample represents sample_weight sessions, so sum the weight.
SELECT date_trunc('minute', sample_time)        AS minute,
       wait_event_component,
       wait_event_class,
       round(sum(sample_weight) / 60.0, 2)      AS avg_active_sessions
FROM yb_active_session_history
WHERE sample_time > now() - INTERVAL '30 minutes'
GROUP BY 1, 2, 3
ORDER BY 1, 4 DESC;

-- Top wait events for one statement during the incident window.
SELECT wait_event_component,
       wait_event,
       wait_event_type,
       sum(sample_weight)                       AS weighted_samples
FROM yb_active_session_history
WHERE query_id = -4127719322                   -- queryid from pg_stat_statements
  AND sample_time BETWEEN '2026-10-03 02:10' AND '2026-10-03 02:40'
GROUP BY 1, 2, 3
ORDER BY 4 DESC;

Then localise. If TServer waits dominate, the tablet prefix in wait_event_aux joined to yb_local_tablets names the table and tablet. A single tablet with most of the samples is a hot shard, which points back to the key design rather than to hardware.

-- YugabyteDB performance monitoring: find the hot tablet behind TServer waits.
-- wait_event_aux holds the first 15 characters of the tablet ID for TServer
-- events; yb_local_tablets maps tablet IDs to tables on THIS node, so run the
-- query on each tserver (or iterate with the collector script further below).
WITH tserver_waits AS (
    SELECT wait_event_aux                AS tablet_prefix,
           wait_event,
           sum(sample_weight)            AS weighted_samples
    FROM yb_active_session_history
    WHERE wait_event_component = 'TServer'
      AND wait_event_aux IS NOT NULL
      AND sample_time > now() - INTERVAL '15 minutes'
    GROUP BY 1, 2
)
SELECT t.namespace_name,
       t.ysql_schema_name,
       t.table_name,
       t.tablet_id,
       w.wait_event,
       w.weighted_samples
FROM tserver_waits AS w
JOIN yb_local_tablets AS t
  ON left(t.tablet_id, 15) = w.tablet_prefix
ORDER BY w.weighted_samples DESC
LIMIT 10;

If one node is busier than the rest, join on top_level_node_id. Skew by node usually means uneven leader placement, uneven client connections, or a node with a slower disk.

-- YugabyteDB performance monitoring: is one node doing more than its share?
-- top_level_node_id joins to the uuid column of yb_servers().
SELECT s.host,
       s.cloud || '/' || s.region || '/' || s.zone  AS placement,
       a.wait_event_class,
       sum(a.sample_weight)                         AS weighted_samples
FROM yb_active_session_history AS a
JOIN yb_servers() AS s
  ON s.uuid = a.top_level_node_id::TEXT
WHERE a.sample_time > now() - INTERVAL '15 minutes'
GROUP BY 1, 2, 3
ORDER BY 4 DESC;

-- Follow one request end to end: every sample that shares its root_request_id,
-- from the YSQL backend down to each tserver RPC it triggered.
SELECT sample_time,
       wait_event_component,
       wait_event,
       wait_event_aux,
       top_level_node_id
FROM yb_active_session_history
WHERE root_request_id = '7f3a0b5e-1d2c-4e8f-9a6b-c3d4e5f6c91e'   -- example value
ORDER BY sample_time;

ASH is per node. For a cluster-wide view, recent releases provide gv$yb_active_session_history; otherwise collect from each node, as the evidence script later in this guide does.

Decoding the wait events

The wait event names are precise once you know the internals behind them. TableRead and IndexRead mean a YSQL backend is waiting on DocDB. Raft_WaitingForReplication means a write is waiting for a majority of replicas. RocksDB_ReadBlockFromFile means a block cache miss turned into a disk read. LockedBatchEntry_Lock and ConflictResolution_WaitOnConflictingTxns are row locks and transaction conflicts. OnCpu_Passive means the work is runnable but waiting for a thread, which is a saturation signal. Figure 3 is the decoder we keep next to every Yugabyte performance troubleshooting session.

Layer 4: live sessions, locks and terminated queries

Live views complete the YugabyteDB observability picture during an incident.

ASH tells you what happened over the last minutes. During an active incident you also need the present tense. YugabyteDB extends pg_stat_activity with allocated_mem_bytes and rss_mem_bytes per backend, extends pg_locks with waitend and a ybdetails JSONB column whose blocked_by attribute names the blocking transaction, and records queries killed by temp_file_limit, SIGSEGV or the OOM killer in yb_terminated_queries (v2024.2 LTS and later).

-- YugabyteDB performance monitoring: what is running, holding or dying right now.

-- 1. Long transactions and leaked "idle in transaction" sessions, with memory.
SELECT pid,
       usename,
       state,
       now() - xact_start                    AS xact_age,
       pg_size_pretty(allocated_mem_bytes)   AS heap,
       pg_size_pretty(rss_mem_bytes)         AS rss,
       left(query, 60)                       AS query
FROM pg_stat_activity
WHERE backend_type = 'client backend'
  AND xact_start < now() - INTERVAL '1 minute'
ORDER BY xact_start;

-- 2. Who is blocked, and by which transaction (distributed lock view).
SET yb_locks_min_txn_age = 5;               -- only transactions older than 5 s
SELECT relation::regclass                    AS table_name,
       mode,
       granted,
       waitend,
       ybdetails ->> 'transactionid'         AS txn_id,
       ybdetails -> 'blocked_by'             AS blocked_by,
       ybdetails ->> 'tablet_id'             AS tablet_id
FROM pg_locks
WHERE NOT granted;

-- 3. Queries killed by temp_file_limit, SIGSEGV or the OOM killer (v2024.2+).
SELECT query_start_time,
       query_end_time,
       termination_reason,
       databasename,
       left(query_text, 80)                  AS query
FROM yb_terminated_queries
ORDER BY query_end_time DESC
LIMIT 20;

Sometimes the right Yugabyte performance troubleshooting move during an incident is to remove one blocking transaction. That is a disruptive action: the application receives an error and must retry. We wrap it in verification, an explicit confirmation gate and validation, never a one-liner.

#!/usr/bin/env bash
# Yugabyte performance troubleshooting, DISRUPTIVE step: cancel a blocking
# distributed transaction found via pg_locks.
# Verification before, operator confirmation gate, validation after.
set -euo pipefail
: "${YB_HOST:?}" "${YB_USER:?}" "${PGPASSWORD:?}" "${TXN_ID:?transaction UUID from ybdetails}"
PSQL=(ysqlsh -h "${YB_HOST}" -p 5433 -U "${YB_USER}" -d yugabyte -At -v ON_ERROR_STOP=1)

# --- Verification BEFORE: is the transaction still there, and who does it block?
"${PSQL[@]}" -c "
  SELECT count(*) AS waiting_on_it
  FROM pg_locks
  WHERE NOT granted
    AND ybdetails -> 'blocked_by' ? '${TXN_ID}';"

read -r -p "Type 'CANCEL ${TXN_ID}' to abort this transaction: " ANSWER
[[ "${ANSWER}" == "CANCEL ${TXN_ID}" ]] || { echo "Aborted, nothing changed."; exit 1; }

# --- Action: the application will receive an error and must retry.
"${PSQL[@]}" -c "SELECT yb_cancel_transaction('${TXN_ID}');"

# --- Validation AFTER: nobody should still be blocked by it.
sleep 2
"${PSQL[@]}" -c "
  SELECT count(*) AS still_waiting_on_it
  FROM pg_locks
  WHERE NOT granted
    AND ybdetails -> 'blocked_by' ? '${TXN_ID}';"

Layer 5: plans and Query Diagnostics for YugabyteDB performance monitoring

Once the evidence points at one statement, prove the hypothesis on an execution plan. EXPLAIN (ANALYZE, DIST) adds Storage Read Requests, Storage Rows Scanned, Storage Read Execution Time and Catalog Read Requests to the plan, so you can see the RPC pattern behind the wait events. Our companion guide on YugabyteDB performance tuning explains how to act on each counter.

-- YugabyteDB performance monitoring: prove the hypothesis on one execution.
EXPLAIN (ANALYZE, DIST, COSTS OFF)
SELECT o.order_id, o.amount
FROM customer_order AS o
WHERE o.customer_id = 4242
  AND o.status = 'OPEN';

-- Counters to compare with what ASH and pg_stat_statements suggested:
--   Storage Read Requests        RPC round trips (should not scale with rows)
--   Storage Rows Scanned         vs rows returned (filter efficiency)
--   Storage Read Execution Time  time waiting on DocDB
--   Catalog Read Requests        cold catalog cache on this backend

For intermittent problems, a single EXPLAIN is not enough. Query Diagnostics (Early Access) captures a bundle for one query_id over an interval: sampled EXPLAIN plans with ANALYZE and DIST, bind variables for slow executions, the matching pg_stat_statements row, the ASH samples and schema details. It needs ysql_yb_enable_query_diagnostics=true on every node.

-- YugabyteDB performance monitoring: capture a full Query Diagnostics bundle
-- (Early Access). Requires yb-tserver flags on every node, then a restart:
--   --ysql_yb_enable_query_diagnostics=true          (default false)
--   --yb_query_diagnostics_circular_buffer_size=64   (KB, default 64)

SELECT yb_query_diagnostics(
    query_id                       => -4127719322,  -- from pg_stat_statements
    diagnostics_interval_sec       => 120,          -- default 300
    explain_sample_rate            => 10,           -- % of executions to EXPLAIN, default 1
    explain_analyze                => true,         -- add actual row counts and timings
    explain_dist                   => true,         -- add DocDB RPC counters
    bind_var_query_min_duration_ms => 50            -- log binds for runs slower than 50 ms
);

-- Bundle lands in pg_data/query_diagnostics/<query_id>/<random-number>/ on that node:
--   constants_and_bind_variables.csv, pg_stat_statements.csv, schema_details.txt,
--   active_session_history.csv, explain_plan.txt
SELECT * FROM yb_query_diagnostics_status;

-- Stop early if you already have what you need.
SELECT yb_cancel_query_diagnostics(query_id => -4127719322);

For a lightweight permanent safety net, the PostgreSQL slow statement log still works in YSQL, passed through the tserver flag that forwards PostgreSQL settings:

# YugabyteDB performance monitoring safety net (yb-tserver flag):
# pass PostgreSQL settings to every YSQL backend.
# Log statements slower than 1 s with their duration; rolling restart to apply.
--ysql_pg_conf_csv="log_min_duration_statement=1000,log_lock_waits=on"

Four incident signatures and how YugabyteDB observability exposes them

Most Yugabyte performance troubleshooting cases we see fall into a handful of patterns. Each leaves a distinct fingerprint across the layers, which is why correlating them matters more than any single chart.

The hot tablet

Signature: p99 rises on one node while others are quiet; ASH shows TServer samples concentrated on one wait_event_aux prefix, often with OnCpu_Active or lock waits; yb_local_tablets resolves it to a single table, frequently one with a monotonically increasing range key. The fix is in the schema, and YugabyteDB performance monitoring has done its job once the tablet and key are named.

The catalog storm after a deploy

Signature: latency jumps right after an application rollout or a burst of DDL; CatalogRead leads the ASH wait list; the CatalogCacheTableMisses rate climbs and yb_ysqlserver_new_connection_total grows faster than usual. New backends start with a cold catalog cache, so connection churn and DDL both show up here. Pooling, including the built-in YSQL Connection Manager, is the usual remedy.

The cross-zone or cross-region write

Signature: write latency rises uniformly on all nodes; Raft_WaitingForReplication dominates ASH for write statements; UpdateConsensus latency follows the network round trip between zones. Nothing is broken; the topology sets the floor. YugabyteDB performance monitoring here feeds a placement decision, such as leader preference or geo-partitioning, rather than a tuning change.

The disk that fell behind

Signature: block cache hit ratio drops, RocksDB_ReadBlockFromFile and WAL_Sync rise in ASH, and log_sync_latency climbs; rocksdb_current_version_num_sst_files and compaction bytes may show compaction debt. The remedy can be capacity, a noisy neighbour, or a query that scans far more than it returns, which docdb_rows_scanned in pg_stat_statements will reveal.

The troubleshooting workflow: from alert to root cause

With the layers in place, Yugabyte performance troubleshooting becomes a repeatable sequence. Each step narrows the search for the next, and each has one primary source, which keeps the team from arguing over dashboards.

1. Confirm. Use PromQL on ports 13000 and 9000 to confirm the regression and find the nodes involved.
2. Rank. Use pg_stat_statements with yb_get_percentile to find the statements that own the extra time.
3. Explain. Group yb_active_session_history by wait event for those query_ids over the incident window.
4. Localise. Use top_level_node_id, wait_event_aux and yb_local_tablets to name the node and tablet.
5. Prove. Reproduce with EXPLAIN (ANALYZE, DIST) or capture a Query Diagnostics bundle.
6. Fix and verify. Change one thing, then rerun the same queries and compare before and after.

Evidence disappears, and YugabyteDB performance monitoring without evidence is guesswork: ASH is a circular buffer and pg_stat_statements may be reset. So the first action when an incident opens is to snapshot everything. This read-only collector gathers metrics twice, 60 seconds apart, plus the per-node SQL views, from every node.

#!/usr/bin/env bash
# YugabyteDB performance monitoring: incident evidence collector (read-only).
# Captures metrics and SQL snapshots from every node into a timestamped folder,
# so the analysis can continue after the incident has cleared.
set -euo pipefail
: "${YB_NODES:?space-separated node list}" "${YB_USER:?}" "${PGPASSWORD:?}"
OUT="yb-evidence-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "${OUT}"

for node in ${YB_NODES}; do
  d="${OUT}/${node}"; mkdir -p "${d}"
  PSQL=(ysqlsh -h "${node}" -p 5433 -U "${YB_USER}" -d yugabyte -v ON_ERROR_STOP=1 --csv)

  # Metric snapshots (two scrapes 60 s apart give you rates offline).
  curl -s "http://${node}:13000/prometheus-metrics" > "${d}/ysql_metrics_t0.prom"
  curl -s "http://${node}:9000/prometheus-metrics?version=v2&table_blocklist=.*" > "${d}/tserver_metrics_t0.prom"

  # Per-node SQL views: ASH, statements, sessions, tablets.
  "${PSQL[@]}" -c "SELECT * FROM yb_active_session_history
                   WHERE sample_time > now() - INTERVAL '60 minutes';" > "${d}/ash.csv"
  "${PSQL[@]}" -c "SELECT * FROM pg_stat_statements;"                   > "${d}/pg_stat_statements.csv"
  "${PSQL[@]}" -c "SELECT * FROM pg_stat_activity;"                     > "${d}/pg_stat_activity.csv"
  "${PSQL[@]}" -c "SELECT * FROM yb_local_tablets;"                     > "${d}/yb_local_tablets.csv"
done

sleep 60
for node in ${YB_NODES}; do
  curl -s "http://${node}:13000/prometheus-metrics" > "${OUT}/${node}/ysql_metrics_t1.prom"
  curl -s "http://${node}:9000/prometheus-metrics?version=v2&table_blocklist=.*" > "${OUT}/${node}/tserver_metrics_t1.prom"
done

# Cluster-wide lock picture once is enough.
ysqlsh -h "${YB_NODES%% *}" -p 5433 -U "${YB_USER}" -d yugabyte --csv \
  -c "SELECT * FROM pg_locks WHERE NOT granted;" > "${OUT}/pg_locks_waiting.csv"

tar czf "${OUT}.tar.gz" "${OUT}" && echo "Evidence: ${OUT}.tar.gz"

Performance Advisor: ASH with a UI

Yugabyte's Performance Advisor is available as a tech preview in YugabyteDB Aeon for clusters on v2024.2 or higher, with YugabyteDB Anywhere support announced as coming. It builds on yb_active_session_history to chart cluster load by wait type, rank queries by their share of load, and flag anomalies such as catalog read waits or lock contention above 50% of wait time. It is a good front end for the same evidence; the SQL in this guide remains the way to verify what it shows and to work on self-managed clusters.

A YugabyteDB performance monitoring dashboard that earns its screen space

Seven panels cover YugabyteDB observability end to end without drowning the on-call engineer.

How MinervaDB runs YugabyteDB performance monitoring

MinervaDB is a vendor-neutral database infrastructure company. Our engineers come from PostgreSQL, MySQL and storage-engine internals, which maps directly onto YSQL, DocDB and RocksDB. For YugabyteDB we design the observability stack, write the alert rules and runbooks, and provide 24×7 consultative support when the alert fires, with response targets of 15 minutes for Severity 1, 12 hours for Severity 2, 24 hours for Severity 3 and 48 hours for Severity 4.

We have supported more than 900 enterprises from 46 cities over 15+ years. Our rule is the one this guide follows: every recommendation is anchored to a named metric, view or wait event, and every change is staged with a rollback path.

Frequently asked questions

What is the best tool for YugabyteDB performance monitoring?

Use Prometheus metrics from ports 13000 and 9000 to detect problems, pg_stat_statements to rank statements, and yb_active_session_history to explain what they were waiting on. No single tool answers all three questions.

How do I find a hot tablet with YugabyteDB observability tools?

Group TServer samples in yb_active_session_history by wait_event_aux, which holds the first 15 characters of the tablet ID, and join it to yb_local_tablets on each node to get the table and tablet.

Does pg_stat_statements work in YugabyteDB?

Yes. It is installed by default and adds yb_latency_histogram, read with yb_get_percentile(), plus DocDB RPC columns such as docdb_read_rpcs, docdb_rows_scanned and docdb_wait_time.

Which ports expose YugabyteDB metrics?

YB-Master on 7000, YB-TServer on 9000, YSQL on 13000 and YCQL on 12000, at /prometheus-metrics for Prometheus format or /metrics for JSON.

How long does Active Session History keep data?

ASH keeps samples in a per-node circular buffer sized by ysql_yb_ash_circular_buffer_size, so retention depends on activity. Export samples during an incident if you need them later.

All SQL, PromQL, configuration and scripts in this post are illustrative and version-pinned to the YugabyteDB releases named; alert thresholds are starting points, not recommendations for your workload. Test every change in a non-production environment first, confirm your exact server version, and maintain a robust disaster-recovery posture, including verified backups and rehearsed failover, before applying anything to production.

Want YugabyteDB observability that finds the root cause before the bridge call ends? Book a session with a MinervaDB principal architect, or email contact@minervadb.com.