Skip to main content
These essential CockroachDB metrics let you monitor your CockroachDB Standard cluster. The metrics are available on graphs on the tabs listed in the Metrics tabs column. Where Custom is listed, the metric may be graphed in a . The Usage column explains why each metric is important to visualize and how to make both practical and actionable use of the metric in a production deployment.

SQL

CockroachDB Metric NameShort NameDescriptionUsageMetrics tab
sql.conn.latencyP90, P99Latency to establish and authenticate a SQL connectionThese metrics characterize the database connection latency which can affect the application performance, for example, by having slow startup times. Connection failures are not recorded in these metrics.
sql.connsConnectionsNumber of open SQL connectionsThis metric shows the number of connections as well as the distribution, or balancing, of connections across cluster nodes. An imbalance can lead to nodes becoming overloaded. Review Connection Pooling.
sql.ddl.countDDL StatementsNumber of SQL DDL statements successfully executedThis high-level metric reflects workload volume. Monitor this metric to identify abnormal application behavior or patterns over time. If abnormal patterns emerge, apply the metric’s time range to the SQL Activity pages to investigate interesting outliers or patterns. For example, on the Transactions page and the Statements page, sort on the Execution Count column. To find problematic sessions, on the Sessions page, sort on the Transaction Count column. Find the sessions with high transaction counts and trace back to a user or application.
sql.delete.countDeleteNumber of SQL DELETE statements successfully executedThis high-level metric reflects workload volume. Monitor this metric to identify abnormal application behavior or patterns over time. If abnormal patterns emerge, apply the metric’s time range to the SQL Activity pages to investigate interesting outliers or patterns. For example, on the Transactions page and the Statements page, sort on the Execution Count column. To find problematic sessions, on the Sessions page, sort on the Transaction Count column. Find the sessions with high transaction counts and trace back to a user or application.,
sql.distsql.contended\_queries.countContentionNumber of SQL queries that experienced contentionThis metric is incremented whenever there is a non-trivial amount of contention experienced by a statement whether read-write or write-write conflicts. Monitor this metric to correlate possible workload performance issues to contention conflicts.
sql.failure.countErrorsNumber of statements resulting in a planning or runtime errorThis metric is a high-level indicator of workload and application degradation with query failures. Use the Insights page to find failed executions with their error code to troubleshoot or use application-level logs, if instrumented, to determine the cause of error.
sql.full.scan.countFull scansNumber of full table or index scansThis metric is a high-level indicator of potentially suboptimal query plans in the workload that may require index tuning and maintenance. To identify the statements with a full table scan, use SHOW FULL TABLE SCAN or the SQL Activity Statements page with the corresponding metric time frame. The Statements page also includes explain plans and index recommendations. Not all full scans are necessarily bad especially over smaller tables.
sql.insert.countInsertNumber of SQL INSERT statements successfully executedThis high-level metric reflects workload volume. Monitor this metric to identify abnormal application behavior or patterns over time. If abnormal patterns emerge, apply the metric’s time range to the SQL Activity pages to investigate interesting outliers or patterns. For example, on the Transactions page and the Statements page, sort on the Execution Count column. To find problematic sessions, on the Sessions page, sort on the Transaction Count column. Find the sessions with high transaction counts and trace back to a user or application.,
sql.new\_connsConnections Per SecondNumber of SQL connections createdThe rate of this metric shows how frequently new connections are being established. This can be useful in determining if a high rate of incoming new connections is causing additional load on the server due to a misconfigured application.,
sql.select.countSelectNumber of SQL SELECT statements successfully executedThis high-level metric reflects workload volume. Monitor this metric to identify abnormal application behavior or patterns over time. If abnormal patterns emerge, apply the metric’s time range to the SQL Activity pages to investigate interesting outliers or patterns. For example, on the Transactions page and the Statements page, sort on the Execution Count column. To find problematic sessions, on the Sessions page, sort on the Transaction Count column. Find the sessions with high transaction counts and trace back to a user or application.,
sql.service.latencyP90, P99, P99.9, P99.99Latency of SQL request executionThese high-level metrics reflect workload performance. Monitor these metrics to understand latency over time. If abnormal patterns emerge, apply the metric’s time range to the SQL Activity pages to investigate interesting outliers or patterns. The Statements page has P90 Latency and P99 latency columns to enable correlation with this metric.,
sql.statements.activeActive StatementsNumber of currently active user SQL statementsThis high-level metric reflects workload volume.
sql.txn.abort.countAbortsNumber of SQL transaction abort errorsThis high-level metric reflects workload performance. A persistently high number of SQL transaction abort errors may negatively impact the workload performance and needs to be investigated.
sql.txn.begin.countBeginNumber of SQL transaction BEGIN statements successfully executedThis metric reflects workload volume by counting explicit transactions. Use this metric to determine whether explicit transactions can be refactored as implicit transactions (individual statements).
sql.txn.commit.countCommitsNumber of SQL transaction COMMIT statements successfully executedThis metric shows the number of transactions that completed successfully. This metric can be used as a proxy to measure the number of successful explicit transactions.
sql.txn.latencyP90, P99Latency of SQL transactionsThese high-level metrics provide a latency histogram of all executed SQL transactions. These metrics provide an overview of the current SQL workload.
sql.txn.rollback.countRollbacksNumber of SQL transaction ROLLBACK statements successfully executedThis metric shows the number of orderly transaction rollbacks. A persistently high number of rollbacks may negatively impact the workload performance and needs to be investigated.
sql.txns.openOpen TransactionsNumber of currently open user SQL transactionsThis metric should roughly correspond to the number of cores * 4. If this metric is consistently larger, scale out the cluster.
sql.update.countUpdateNumber of SQL UPDATE statements successfully executedThis high-level metric reflects workload volume. Monitor this metric to identify abnormal application behavior or patterns over time. If abnormal patterns emerge, apply the metric’s time range to the SQL Activity pages to investigate interesting outliers or patterns. For example, on the Transactions page and the Statements page, sort on the Execution Count column. To find problematic sessions, on the Sessions page, sort on the Transaction Count column. Find the sessions with high transaction counts and trace back to a user or application.,
txn.restarts.asyncwritefailureAsync Consensus FailureNumber of restarts due to async consensus writes that failed to leave intentsThis metric is one measure of the impact of contention conflicts on workload performance. For guidance on contention conflicts, review transaction contention best practices and performance tuning recipes. Tens of restarts per minute may be a high value, a signal of an elevated degree of contention in the workload, which should be investigated. For the specific error, refer to the transaction retry error reference for more details.
txn.restarts.readwithinuncertaintyRead Within Uncertainty IntervalNumber of restarts due to reading a new value within the uncertainty intervalThis metric is one measure of the impact of contention conflicts on workload performance. For guidance on contention conflicts, review transaction contention best practices and performance tuning recipes. Tens of restarts per minute may be a high value, a signal of an elevated degree of contention in the workload, which should be investigated. For the specific error, refer to the transaction retry error reference for more details.
txn.restarts.serializableForwarded TimestampNumber of restarts due to a forwarded commit timestamp and isolation=SERIALIZABLEThis metric is one measure of the impact of contention conflicts on workload performance. For guidance on contention conflicts, review transaction contention best practices and performance tuning recipes. Tens of restarts per minute may be a high value, a signal of an elevated degree of contention in the workload, which should be investigated.
txn.restarts.txnabortedAbortedNumber of restarts due to an abort by a concurrent transaction (usually due to deadlock)The errors tracked by this metric are generally due to deadlocks. Deadlocks can often be prevented with a considered transaction design. Identify the conflicting transactions involved in the deadlocks, then, if possible, redesign the business logic implementation prone to deadlocks.
txn.restarts.txnpushPush FailureNumber of restarts due to a transaction push failureThis metric is one measure of the impact of contention conflicts on workload performance. For guidance on contention conflicts, review transaction contention best practices and performance tuning recipes. Tens of restarts per minute may be a high value, a signal of an elevated degree of contention in the workload, which should be investigated.
txn.restarts.unknownUnknownNumber of restarts due to a unknown reasonsThis metric is one measure of the impact of contention conflicts on workload performance. For guidance on contention conflicts, review transaction contention best practices and performance tuning recipes. Tens of restarts per minute may be a high value, a signal of an elevated degree of contention in the workload, which should be investigated.
txn.restarts.writetoooldWrite Too OldNumber of restarts due to a concurrent writer committing firstThis metric is one measure of the impact of contention conflicts on workload performance. For guidance on contention conflicts, review transaction contention best practices and performance tuning recipes. Tens of restarts per minute may be a high value, a signal of an elevated degree of contention in the workload, which should be investigated.
txn.restarts.writetoooldmultiWrite Too Old (multiple)This metric is one measure of the impact of contention conflicts on workload performance. For guidance on contention conflicts, review transaction contention best practices and performance tuning recipes. Tens of restarts per minute may be a high value, a signal of an elevated degree of contention in the workload, which should be investigated. For the specific error, refer to the transaction retry error reference for more details.

Storage

CockroachDB Metric NameShort NameDescriptionUsageMetrics tab
livebytesStorage usageNumber of bytes of live data (keys plus values)The amount of data being stored in the cluster. In the Storage Used graph, this is the logical number of live bytes and does not account for compression or replication.

Table Statistics

CockroachDB Metric NameShort NameDescriptionUsageMetrics tab
jobs.auto\_create\_stats.currently\_pausedAuto Create Statistics PausedNumber of auto_create_stats jobs currently considered PausedThis metric is a high-level indicator that automatically generated statistics jobs are paused which can lead to the query optimizer running with stale statistics. Stale statistics can cause suboptimal query plans to be selected leading to poor query performance.,
jobs.auto\_create\_stats.currently\_runningAuto Create Statistics RunningNumber of auto_create_stats jobs currently running in Resume or OnFailOrCancel stateThis metric tracks the number of active automatically generated statistics jobs that could also be consuming resources. Ensure that foreground SQL traffic is not impacted by correlating this metric with SQL latency and query volume metrics.,
jobs.auto\_create\_stats.resume\_failedAuto Create Statistics FailedNumber of auto_create_stats jobs which failed with a non-retriable errorThis metric is a high-level indicator that automatically generated table statistics is failing. Failed statistic creation can lead to the query optimizer running with stale statistics. Stale statistics can cause suboptimal query plans to be selected leading to poor query performance.,
jobs.create\_stats.currently\_runningCreate Statistics RunningNumber of create_stats jobs currently running in Resume or OnFailOrCancel stateThis metric tracks the number of active create statistics jobs that could also be consuming resources. Ensure that foreground SQL traffic is not impacted by correlating this metric with SQL latency and query volume metrics.,

Changefeeds

CockroachDB Metric NameShort NameDescriptionUsageMetrics tab
changefeed.commit\_latencyP99, P90Event commit latency: a difference between event MVCC timestamp and the time it was acknowledged by the downstream sink. If the sink batches events, then the difference between the oldest event in the batch and acknowledgement is recorded; Excludes latency during backfillThis metric provides a useful context when assessing the state of changefeeds. This metric characterizes the end-to-end lag between a committed change and that change applied at the destination.,
changefeed.emitted\_bytesEmitted bytesBytes emitted by all feedsThis metric provides a useful context when assessing the state of changefeeds. This metric characterizes the throughput bytes being streamed from the CockroachDB cluster.,
changefeed.emitted\_messagesEmitted messagesMessages emitted by all feedsThis metric provides a useful context when assessing the state of changefeeds. This metric characterizes the rate of changes being streamed from the CockroachDB cluster.,
changefeed.error\_retriesErrorsTotal retryable errors encountered by all changefeedsThis metric tracks transient changefeed errors. Alert on “too many” errors, such as 50 retries in 15 minutes. For example, during a rolling upgrade this counter will increase because the changefeed jobs will restart following node restarts. There is an exponential backoff, up to 10 minutes. But if there is no rolling upgrade in process or other cluster maintenance, and the error rate is high, investigate the changefeed job.,
changefeed.failuresFailuresTotal number of changefeed jobs which have failedThis metric tracks the permanent changefeed job failures that the jobs system will not try to restart. Any increase in this counter should be investigated. An alert on this metric is recommended.,
changefeed.runningRunningNumber of currently running changefeeds, including sinklessThis metric tracks the total number of all running changefeeds.,
jobs.changefeed.currently\_pausedPausedNumber of changefeed jobs currently considered PausedMonitor and alert on this metric to safeguard against an inadvertent operational error of leaving a changefeed job in a paused state for an extended period of time. Changefeed jobs should not be paused for a long time because the protected timestamp prevents garbage collection.,
jobs.changefeed.protected\_age\_secProtected Timestamp AgeThe age of the oldest PTS record protected by changefeed jobsChangefeeds use protected timestamps to protect the data from being garbage collected. Ensure the protected timestamp age does not significantly exceed the GC TTL zone configuration. Alert on this metric if the protected timestamp age is greater than 3 times the GC TTL.,

Row-Level TTL

CockroachDB Metric NameShort NameDescriptionUsageMetrics tab
jobs.row\_level\_ttl.currently\_pausedPausedNumber of row_level_ttl jobs currently considered PausedMonitor this metric to ensure the Row Level TTL job does not remain paused inadvertently for an extended period.,
jobs.row\_level\_ttl.currently\_runningRunningNumber of row_level_ttl jobs currently running in Resume or OnFailOrCancel stateMonitor this metric to ensure there are not too many Row Level TTL jobs running at the same time. Generally, this metric should be in the low single digits.,
jobs.row\_level\_ttl.delete\_durationDelete Latency (P90), Delete Latency (P99)Duration for delete requests during row level TTL.See Description.
jobs.row\_level\_ttl.resume\_completedResume CompletedNumber of row_level_ttl jobs which successfully resumed to completionIf Row Level TTL is enabled, this metric should be nonzero and correspond to the ttl_cron setting that was chosen. If this metric is zero, it means the job is not running,
jobs.row\_level\_ttl.resume\_failedResume FailedNumber of row_level_ttl jobs which failed with a non-retriable errorThis metric should remain at zero. Repeated errors means the Row Level TTL job is not deleting data.,
jobs.row\_level\_ttl.rows\_deletedRows deletedNumber of rows deleted by the row level TTL job.Correlate this metric with the metric jobs.row_level_ttl.rows_selected to ensure all the rows that should be deleted are actually getting deleted.,
jobs.row\_level\_ttl.rows\_selectedRows selectedNumber of rows selected for deletion by the row level TTL job.Correlate this metric with the metric jobs.row_level_ttl.rows_deleted to ensure all the rows that should be deleted are actually getting deleted.,
jobs.row\_level\_ttl.select\_durationSelect Latency (P90), Select Latency (P99)Duration for select requests during row level TTL.See Description.
jobs.row\_level\_ttl.total\_expired\_rowsExpired RowsApproximate number of rows that have expired the TTL on the TTL table.See Description.
jobs.row\_level\_ttl.total\_rowsTotal RowsApproximate number of rows on the TTL table.See Description.
schedules.scheduled-row-level-ttl-executor.failedFailed SchedulesNumber of scheduled-row-level-ttl-executor jobs failedMonitor this metric to ensure the Row Level TTL job is running. If it is non-zero, it means the job could not be created.,

See also