> ## Documentation Index
> Fetch the complete documentation index at: https://cockroachlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# SQL Dashboard

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

The **SQL** dashboard in the DB Console lets you monitor the performance of your SQL queries.

To view this dashboard, <InternalLink path="ui-overview#db-console-access">access the DB Console</InternalLink>, click **Metrics** in the left-hand navigation, and then select **Dashboard** > **SQL**.

## Dashboard navigation

Use the **Graph** menu to display metrics for your entire cluster or for a specific node.

To the right of the Graph and Dashboard menus, a time interval selector allows you to filter the view for a predefined or custom time interval. Use the navigation buttons to move to the previous, next, or current time interval. When you select a time interval, the same interval is selected in the <InternalLink path="ui-overview#sql-activity">SQL Activity</InternalLink> pages. However, if you select 10 or 30 minutes, the interval defaults to 1 hour in SQL Activity pages.

When viewing graphs, a tooltip will appear at your mouse cursor providing further insight into the data under the mouse cursor. Click anywhere within the graph to pin the tooltip in place, decoupling the tooltip from your mouse movements. Click anywhere within the graph to cause the tooltip to follow your mouse once more.

For monitoring CockroachDB, it is sufficient to use the [**Open SQL Sessions**](#open-sql-sessions), [**SQL Byte Traffic**](#sql-byte-traffic), [**SQL Statements**](#sql-statements), [**Service Latency**](#service-latency-sql-99th-percentile), and [**Transactions**](#transactions) graphs.

The **SQL** dashboard displays the following time series graphs:

## Open SQL Sessions

* In the node view, the graph shows the number of connections open between the client and the selected node.

  * To control the maximum number of non-superuser (<InternalLink path="security-reference/authorization">`root`</InternalLink> user or other <InternalLink path="security-reference/authorization#admin-role">`admin` role</InternalLink>) connections a <InternalLink path="architecture/sql-layer">gateway node</InternalLink> can have open at one time, use the `server.max_connections_per_gateway` <InternalLink path="cluster-settings">cluster setting</InternalLink>.  If a new non-superuser connection would exceed this limit, the error message `"sorry, too many clients already"` is returned, along with error code `53300`.

* In the cluster view, the graph shows the total number of SQL client connections to all nodes combined, with lines for each node.

## SQL Connection Rate

The **SQL Connection Rate** is an average of the number of connection attempts per second over an aggregation window.

* In the node view, the graph shows the rate of SQL connection attempts between clients and the selected node.

* In the cluster view, the graph shows the rate of SQL connection attempts to all nodes, with lines for each node.

## Open SQL Transactions

* In the node view, the graph shows the total number of open SQL transactions on the node.

* In the cluster view, the graph shows the total number of open SQL transactions across all nodes in the cluster.

See the <InternalLink path="ui-transactions-page">Transactions page</InternalLink> for more details on the transactions.

## Active SQL Statements

* In the node view, the graph shows the total number of SQL statements running on that node.

* In the cluster view, the graph shows the total number of SQL statements running across all nodes in the cluster.

See the <InternalLink path="ui-statements-page">Statements page</InternalLink> for more details on the cluster's SQL statements.

## SQL Byte Traffic

The **SQL Byte Traffic** graph helps you correlate SQL query count to byte traffic, especially in bulk data inserts or analytic queries that return data in bulk.

* In the node view, the graph shows the current byte throughput (bytes/second) between all the connected SQL clients and the node. There are lines for bytes in and bytes out.

* In the cluster view, the graph shows the aggregate client throughput across all nodes. There are lines for bytes in and bytes out.

## SQL Statements

* In the node view, the graph shows the 10-second average of the number of `SELECT`/`INSERT`/`UPDATE`/`DELETE` statements per second issued by SQL clients on the node.

* In the cluster view, the graph shows the sum of the per-node averages, that is, an aggregate estimation of the current statement load over the cluster, assuming the last 10 seconds of activity per node are representative of this load.

See the <InternalLink path="ui-statements-page">Statements page</InternalLink> for more details on the cluster's SQL statements.

Metrics: `sql.select.count`, `sql.update.count`, `sql.insert.count`, `sql.delete.count`

The following SQL statements update the `INSERT` metric (`sql.insert.count`):

* <InternalLink path="insert#on-conflict-clause">`INSERT ... ON CONFLICT DO UPDATE ...`</InternalLink>: Even when the `DO UPDATE` clause is actually executed, the root of the <InternalLink path="architecture/sql-layer#parsing">abstract syntax tree (AST)</InternalLink> is used to increment the metric, rather than the actual execution details.

* <InternalLink path="upsert">`UPSERT`</InternalLink>

<Note>
  <InternalLink path="sql-statements#data-manipulation-statements">Data manipulation statements</InternalLink> other than  `SELECT`/`INSERT`/`UPDATE`/`DELETE`/`UPSERT` update the `sql.misc.count` metric, which is *not* displayed on this graph.
</Note>

## SQL Statement Errors

* In the node view, the graph shows the 10-second average of the number of SQL statements issued to the node that returned a <InternalLink path="architecture/sql-layer#sql-parser-planner-executor">planning</InternalLink>,  <InternalLink path="architecture/sql-layer#sql-parser-planner-executor">runtime</InternalLink>, or <InternalLink path="transactions#error-handling">retry error</InternalLink>.

* In the cluster view, the graph shows the 10-second average of the number of SQL statements that returned a <InternalLink path="architecture/sql-layer#sql-parser-planner-executor">planning</InternalLink>,  <InternalLink path="architecture/sql-layer#sql-parser-planner-executor">runtime</InternalLink>, or <InternalLink path="transactions#error-handling">retry error</InternalLink> across all nodes.

See the <InternalLink path="ui-statements-page">Statements page</InternalLink> for more details on the cluster's SQL statements.

## SQL Statement Contention

The statement contention metric is a counter that represents the number of statements that have experienced <InternalLink path="performance-best-practices-overview#transaction-contention">contention</InternalLink>. If a statement experiences at least one contention "event" (i.e., the statement is forced to wait for another transaction), the counter is incremented at most once.

* In the node view, the graph shows the total number of SQL statements that experienced <InternalLink path="performance-best-practices-overview#transaction-contention">contention</InternalLink> on that node.

* In the cluster view, the graph shows the total number of SQL statements that experienced <InternalLink path="performance-best-practices-overview#transaction-contention">contention</InternalLink> across all nodes in the cluster.

  See the <InternalLink path="ui-statements-page">Statements page</InternalLink> for more details on the cluster's SQL statements.

## Full Table/Index Scans

* In the node view, the graph shows the total number of full table and index scans on that node.

* In the cluster view, the graph shows the total number of full table and index scans across all nodes in the cluster.

<InternalLink path="sql-tuning-with-explain">Examine the statements</InternalLink> that result in full table scans and consider adding <InternalLink path="schema-design-indexes#create-a-secondary-index">secondary indexes</InternalLink>.

## Active Flows for Distributed SQL Statements

* In the node view, the graph shows the number of flows on that node contributing to the running <InternalLink path="architecture/sql-layer#distsql">distributed SQL</InternalLink> statements.

* In the cluster view, the graph shows the number of flows across all nodes in the cluster contributing to the running <InternalLink path="architecture/sql-layer#distsql">distributed SQL</InternalLink> statements.

## Connection Latency: 99th Percentile

Connection latency is calculated as the time in nanoseconds between when the cluster receives a connection request and establishes the connection to the client, including authentication. This metric characterizes the database connection latency which can affect the application performance, for example, by having slow startup times. Connection failures are not recorded in these metrics.

* In the node view, the graph shows the 99th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of connection latency for the node. Over the last minute this node established 99% of connections within this time, not including network latency between the node and the client.

* In the cluster view, the graph shows the 99th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of connection latency across all nodes in the cluster. There are lines for each node in the cluster. Over the last minute the nodes of the cluster established 99% of connections within this time, not including network latency between the node and the client.

Metrics: `sql.conn.latency-p99`

## Connection Latency: 90th Percentile

Connection latency is calculated as the time in nanoseconds between when the cluster receives a connection request and establishes the connection to the client, including authentication. This metric characterizes the database connection latency which can affect the application performance, for example, by having slow startup times. Connection failures are not recorded in these metrics.

* In the node view, the graph shows the 90th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of connection latency for the node. Over the last minute this node established 90% of connections within this time, not including network latency between the node and the client.

* In the cluster view, the graph shows the 90th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of connection latency across all nodes in the cluster. There are lines for each node in the cluster. Over the last minute the nodes of the cluster established 90% of connections within this time, not including network latency between the node and the client.

Metrics: `sql.conn.latency-p90`

## Service Latency: SQL, 99.99th percentile

Service latency is calculated as the time in nanoseconds between when the cluster <InternalLink path="architecture/sql-layer">receives a query and finishes executing the query</InternalLink>. This time does not include returning results to the client. Service latency includes metrics only from DML  (`SELECT`,` INSERT`, `UPDATE`, and `DELETE`) statements.

* In the node view, the graph shows the 99.99th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of service latency for the node. Over the last minute this node executed 99.99% of queries within this time, not including network latency between the node and the client.

* In the cluster view, the graph shows the 99.99th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of service latency across all nodes in the cluster. There are lines for each node in the cluster. Over the last minute the cluster executed 99.99% of queries within this time, not including network latency between the node and the client.

## Service Latency: SQL, 99.9th percentile

Service latency is calculated as the time in nanoseconds between when the cluster <InternalLink path="architecture/sql-layer">receives a query and finishes executing the query</InternalLink>. This time does not include returning results to the client. Service latency includes metrics only from DML  (`SELECT`,` INSERT`, `UPDATE`, and `DELETE`) statements.

* In the node view, the graph shows the 99.9th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of service latency for the node. Over the last minute this node executed 99.9% of queries within this time, not including network latency between the node and the client.

* In the cluster view, the graph shows the 99.9th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of service latency across all nodes in the cluster. There are lines for each node in the cluster. Over the last minute the cluster executed 99.9% of queries within this time, not including network latency between the node and the client.

## Service Latency: SQL, 99th percentile

Service latency is calculated as the time in nanoseconds between when the cluster <InternalLink path="architecture/sql-layer">receives a query and finishes executing the query</InternalLink>. This time does not include returning results to the client. Service latency includes metrics only from DML  (`SELECT`,` INSERT`, `UPDATE`, and `DELETE`) statements.

* In the node view, the graph shows the 99th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of service latency for the node. Over the last minute this node executed 99% of queries within this time, not including network latency between the node and the client.

* In the cluster view, the graph shows the 99th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of service latency across all nodes in the cluster. There are lines for each node in the cluster. Over the last minute the node executed 99% of queries within this time, not including network latency between the node and the client.

## Service Latency: SQL, 90th percentile

Service latency is calculated as the time in nanoseconds between when the cluster <InternalLink path="architecture/sql-layer">receives a query and finishes executing the query</InternalLink>. This time does not include returning results to the client. Service latency includes metrics only from DML  (`SELECT`,` INSERT`, `UPDATE`, and `DELETE`) statements.

* In the node view, the graph shows the 90th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of service latency for the node. Over the last minute this node executed 90% of queries within this time, not including network latency between the node and the client.

* In the cluster view, the graph shows the 90th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of service latency across all nodes in the cluster. There are lines for each node in the cluster. Over the last minute the cluster executed 90% of queries within this time, not including network latency between the node and the client.

## KV Execution Latency: 99th percentile

KV execution latency is calculated as the time in milliseconds between when the <InternalLink path="architecture/overview">KV layer</InternalLink> receives the request and delivers a response.

* In the node view, the graph shows the 99th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of KV execution latency for the node. Over the last minute the node executed 99% of requests within this time.

* In the cluster view, the graph shows the 99th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of KV execution latency for each node in the cluster. There are lines for each node in the cluster. Over the last minute the node executed 99% of requests within this time.

## KV Execution Latency: 90th percentile

KV execution latency is calculated as the time in milliseconds between when the <InternalLink path="architecture/overview">KV layer</InternalLink> receives the request and delivers a response.

* In the node view, the graph shows the 90th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of KV execution latency for the node. Over the last minute the node executed 90% of requests within this time.

* In the cluster view, the graph shows the 90th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of KV execution latency for each node in the cluster. There are lines for each node in the cluster. Over the last minute the node executed 90% of requests within this time.

## Transactions

* In the node view, the graph shows the 10-second average of the number of opened (`Begin`), committed (`Commits`), rolled back (`Rollbacks`), and aborted (`Aborts`) <InternalLink path="transactions">transactions</InternalLink> per second issued by SQL clients on the node.

* In the cluster view, the graph shows the sum of the per-node averages, that is, an aggregate estimation of the current <InternalLink path="transactions">transactions</InternalLink> load over the cluster, assuming the last 10 seconds of activity per node are representative of this load.

If the graph shows excessive aborts or rollbacks, it might indicate issues with the SQL statements. In that case, re-examine <InternalLink path="ui-statements-page">statements</InternalLink> to lower contention.

See the <InternalLink path="ui-transactions-page">Transactions page</InternalLink> for more details on the transactions.

## Transaction Restarts

* In the node view, the graph shows the number of transactions restarted on that node broken down by the errors that caused the restart.

* In the cluster view, the graph shows the number of transactions restarted across the cluster broken down by the errors that caused the restart.

See the <InternalLink path="transaction-retry-error-reference">Transaction Retry Error Reference</InternalLink> for details on the errors that caused the transaction to restart.

## Transaction Latency: 99th percentile

Transaction latency is calculated as the total time in nanoseconds a <InternalLink path="transactions">transaction</InternalLink> took to complete.

* In the node view, the graph shows the 99th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of transaction time over a 1 minute period for the node. Over the last minute the node completed 99% of transactions within this time.

* In the cluster view, the graph shows the 99th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of transaction time over a 1 minute period for each node in the cluster. Over the last minute the node completed 99% of transactions within this time.

See the <InternalLink path="ui-transactions-page">Transactions page</InternalLink> for more details on the transactions.

## Transaction Latency: 90th percentile

Transaction latency is calculated as the total time in nanoseconds a <InternalLink path="transactions">transaction</InternalLink> took to complete.

* In the node view, the graph shows the 90th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of transaction time over a 1 minute period for the node. Over the last minute the node completed 90% of transactions within this time.

* In the cluster view, the graph shows the 90th [percentile](https://wikipedia.org/wiki/Percentile#The_normal_distribution_and_percentiles) of transaction time over a 1 minute period for each node in the cluster. Over the last minute the node completed 90% of transactions within this time.

See the <InternalLink path="ui-transactions-page">Transactions page</InternalLink> for more details on the transactions.

## SQL Memory

* In the node view, the graph shows the current amount of memory in KiB allocated to the SQL layer on this node. This amount is what is compared against the node's <InternalLink path="cockroach-start#general">`--max-sql-memory` flag</InternalLink>.

* In the cluster view, the graph shows the current amount of memory in KiB allocated to the SQL layer on all nodes in the cluster. This amount is what is compared against the node's <InternalLink path="cockroach-start#general">`--max-sql-memory` flag</InternalLink>.

**Expected values for a healthy cluster**: This value should not exceed the <InternalLink path="recommended-production-settings#cache-and-sql-memory-size">`--max-sql-memory`</InternalLink> size. A healthy threshold is 75% of allocated `--max-sql-memory`.

## Schema Changes

* In the node view, the graph shows the total number of <InternalLink path="online-schema-changes">DDL statements</InternalLink> per second on the node.

* In the cluster view, the graph shows the total number of <InternalLink path="online-schema-changes">DDL statements</InternalLink> per second across all nodes in the cluster.

## Statement Denials: Cluster Settings

Statement denials are statements that were denied due to a <InternalLink path="cluster-settings">cluster setting</InternalLink> with the following format:

```
{feature}.{statement_type}.enabled = FALSE
```

* In the node view, the graph shows the total number of statements denied per second on this node.

* In the cluster view, the graph shows the total number of statements denied per second across all nodes in the cluster.

## Summary and events

### Summary panel

A **Summary** panel of key metrics is displayed to the right of the timeseries graphs.

| Metric             | Description                                                                                                                                                            |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Total Nodes        | The total number of nodes in the cluster. [Decommissioned nodes](node-shutdown.html?filters=decommission) are not included in this count.                              |
| Capacity Used      | The storage capacity used as a percentage of <InternalLink path="ui-cluster-overview-page#capacity-metrics">usable capacity</InternalLink> allocated across all nodes. |
| Unavailable Ranges | The number of unavailable ranges in the cluster. A non-zero number indicates an unstable cluster.                                                                      |
| Queries per second | The total number of `SELECT`, `UPDATE`, `INSERT`, and `DELETE` queries executed per second across the cluster.                                                         |
| P99 Latency        | The 99th percentile of service latency.                                                                                                                                |

If you are testing your deployment locally with multiple CockroachDB nodes running on a single machine (this is <InternalLink path="recommended-production-settings#topology">not recommended in production</InternalLink>), you must explicitly <InternalLink path="cockroach-start#store">set the store size</InternalLink> per node in order to display the correct capacity. Otherwise, the machine's actual disk capacity will be counted as a separate store for each node, thus inflating the computed capacity.

### Events panel

Underneath the [Summary](#summary-panel) panel, the **Events** panel lists the 5 most recent events logged for all nodes across the cluster. To list all events, click **View all events**.

<img src="https://mintcdn.com/cockroachlabs/DqKzx0qjg1R7yDTM/images/v23.1/ui_events.png?fit=max&auto=format&n=DqKzx0qjg1R7yDTM&q=85&s=8c488199f04bf8fee057db7c20298211" alt="DB Console Events" width="668" height="1036" data-path="images/v23.1/ui_events.png" />

The following types of events are listed:

* Database created
* Database dropped
* Table created
* Table dropped
* Table altered
* Index created
* Index dropped
* View created
* View dropped
* Schema change reversed
* Schema change finished
* Node joined
* Node decommissioned
* Node restarted
* Cluster setting changed

## See also

* <InternalLink path="troubleshooting-overview">Troubleshooting Overview</InternalLink>
* <InternalLink path="support-resources">Support Resources</InternalLink>
* <InternalLink path="monitoring-and-alerting#raw-status-endpoints">Raw Status Endpoints</InternalLink>