Skip to main content
Multi-dimensional metrics are additional time series with extra labels. This page will help you understand the potential size of the Prometheus scrape payload for your workload when multi-dimensional metrics are enabled. The number of multi-dimensional metrics can significantly increase based on their associated labels, which increases cardinality. The export of multi-dimensional metrics can be enabled by two :

​
Enable child metrics

Child metrics are specific, detailed metrics that are usually related to a higher-level (parent or aggregate) metric. They often provide more granular or specific information about a particular aspect of the parent metric. The parent metrics and their potential child metrics are determined by the specific feature the cluster is using. The is disabled by default. To enable it, use the statement.
The (default: true) reports an aggregate time series for applicable multi-dimensional metrics. When set to false, it stops reporting the aggregate time series, preventing double counting when querying those metrics.

​
All clusters

An RPC (Remote Procedure Call) connection is a communication method used in distributed systems, like CockroachDB, to allow one program to request a service from a program located in another computer on a network without having to understand the network’s details. In the context of CockroachDB, RPC connections are used for inter-node communication. For instance, if Node 1 sends a request to Node 2, and Node 2 dials back (sends request back to Node 1), it ensures that communication is healthy in both directions. This is referred to as a “bidirectionally connected” and “heartbeating” RPC connection. When child metrics is enabled, for all clusters the rpc.connection.* metrics are exported per-peer with labels for remote_node_id, remote_addr, and class. The class label may have the following values: system, default, and raft. The cardinality increases with the number of nodes. An aggregated metric is also included. For example:
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
rpc.connection.avg_round_trip_latency
Sum of exponentially weighted moving average of round-trip latencies, as measured through a gRPC RPC. Since this metric is based on gRPC RPCs, it is affected by application-level processing delays and CPU overload effects. See rpc.connection.tcp_rtt for a metric that is obtained from the kernel’s TCP stack. Dividing this Gauge by rpc.connection.healthy gives an approximation of average latency, but the top-level round-trip-latency histogram is more useful. Instead, users should consult the label families of this metric if they are available (which requires prometheus and the cluster setting ‘server.child_metrics.enabled’); these provide per-peer moving averages. This metric does not track failed connection. A failed connection’s contribution is reset to zero.GAUGENANOSECONDS
rpc.connection.failures
Counter of failed connections. This includes both the event in which a healthy connection terminates as well as unsuccessful reconnection attempts. Connections that are terminated as part of local node shutdown are excluded. Decommissioned peers are excluded.COUNTERCOUNT
rpc.connection.healthy
Gauge of current connections in a healthy state (i.e. bidirectionally connected and heartbeating)GAUGECOUNT
rpc.connection.healthy_nanos
Gauge of nanoseconds of healthy connection time On the prometheus endpoint scraped with the cluster setting ‘server.child_metrics.enabled’ set, the constituent parts of this metric are available on a per-peer basis and one can read off for how long a given peer has been connectedGAUGENANOSECONDS
rpc.connection.heartbeats
Counter of successful heartbeats.COUNTERCOUNT
rpc.connection.inactive
Gauge of current connections in an inactive state and pending deletion; these are not healthy but are not tracked as unhealthy either because there is reason to believe that the connection is no longer relevant,for example if the node has since been seen under a new addressGAUGECOUNT
rpc.connection.unhealthy
Gauge of current connections in an unhealthy state (not bidirectionally connected or heartbeating)GAUGECOUNT
rpc.connection.unhealthy_nanos
Gauge of nanoseconds of unhealthy connection time. On the prometheus endpoint scraped with the cluster setting ‘server.child_metrics.enabled’ set, the constituent parts of this metric are available on a per-peer basis and one can read off for how long a given peer has been unreachableGAUGENANOSECONDS

​
Secure clusters

When child metrics is enabled, for the security.certificate.expiration.client is exported per SQL user with a label for sql_user. The sql_user label may have the values of the cluster’s users who are logged into a node using client security certificates. The cardinality increases with the number of SQL users. An aggregated metric is also included, however since this is a sum of the child metric values which represent timestamps, it is not usable. For example:
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
security.certificate.expiration.client
Minimum expiration for client certificates, labeled by SQL user. 0 means no certificate or error.GAUGESECONDS

​
Virtual clusters

When child metrics is enabled, for the kv.tenant_rate_limit.* metrics and other kv-related metrics are exported per virtual cluster with a label for tenant_id. The tenant_id label may have the values: system or the id of the virtual cluster. The cardinality increases with the number of virtual clusters. An aggregated metric is also included.
With virtual clusters, while the tenant_id label on kv metrics is only exported when child metrics is enabled, is exported whether child metrics is enabled or disabled.
For example:
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
abortspanbytes
Number of bytes in the abort spanGAUGEBYTES
gcbytesage
Cumulative age of non-live dataGAUGESECONDS
intentage
Cumulative age of locksGAUGESECONDS
intentbytes
Number of bytes in intent KV pairsGAUGEBYTES
intentcount
Count of intent keysGAUGECOUNT
keybytes
Number of bytes taken up by keysGAUGEBYTES
keycount
Count of all keysGAUGECOUNT
kv.tenant_rate_limit.current_blocked
Number of requests currently blocked by the rate limiterGAUGECOUNT
kv.tenant_rate_limit.num_tenants
Number of tenants currently being trackedGAUGECOUNT
kv.tenant_rate_limit.read_batches_admitted
Number of read batches admitted by the rate limiterCOUNTERCOUNT
kv.tenant_rate_limit.read_bytes_admitted
Number of read bytes admitted by the rate limiterCOUNTERBYTES
kv.tenant_rate_limit.read_requests_admitted
Number of read requests admitted by the rate limiterCOUNTERCOUNT
kv.tenant_rate_limit.write_batches_admitted
Number of write batches admitted by the rate limiterCOUNTERCOUNT
kv.tenant_rate_limit.write_bytes_admitted
Number of write bytes admitted by the rate limiterCOUNTERBYTES
kv.tenant_rate_limit.write_requests_admitted
Number of write requests admitted by the rate limiterCOUNTERCOUNT
livebytes
Number of bytes of live data (keys plus values)GAUGEBYTES
livecount
Count of live keysGAUGECOUNT
lockbytes
Number of bytes taken up by replicated lock key-values (shared and exclusive strength, not intent strength)GAUGEBYTES
lockcount
Count of replicated locks (shared, exclusive, and intent strength)GAUGECOUNT
rangekeybytes
Number of bytes taken up by range keys (e.g. MVCC range tombstones)GAUGEBYTES
rangekeycount
Count of all range keys (e.g. MVCC range tombstones)GAUGECOUNT
rangevalbytes
Number of bytes taken up by range key values (e.g. MVCC range tombstones)GAUGEBYTES
rangevalcount
Count of all range key values (e.g. MVCC range tombstones)GAUGECOUNT
sysbytes
Number of bytes in system KV pairsGAUGEBYTES
syscount
Count of system KV pairsGAUGECOUNT
totalbytes
Total number of bytes taken up by keys and values including non-live dataGAUGEBYTES
valbytes
Number of bytes taken up by valuesGAUGEBYTES
valcount
Count of all valuesGAUGECOUNT

​
Clusters with changefeeds

When child metrics is enabled and are created on the cluster, the changefeed.* metrics are exported per changefeed metric label with a label for scope. The scope label may have the values set using the metrics_label option. The cardinality increases with the number of changefeed metric labels. An aggregated metric is also included. For example, when you create two changefeeds with the metrics labels employees and office_dogs, the counter metric changefeed_error_retries exports child metrics with a scope for employees and office_dogs. In addition, the default scope will also be exported which includes changefeeds started without a metrics label.
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
changefeed.admit_latency
Event admission latency: a difference between event MVCC timestamp and the time it was admitted into changefeed pipeline; Note: this metric includes the time spent waiting until event can be processed due to backpressure or time spent resolving schema descriptors. Also note, this metric excludes latency during backfillHISTOGRAMNANOSECONDS
changefeed.aggregator_progress
The earliest timestamp up to which any aggregator is guaranteed to have emitted all values forGAUGETIMESTAMP_NS
changefeed.backfill_count
Number of changefeeds currently executing backfillGAUGECOUNT
changefeed.backfill_pending_ranges
Number of ranges in an ongoing backfill that are yet to be fully emittedGAUGECOUNT
changefeed.batch_reduction_count
Number of times a changefeed aggregator node attempted to reduce the size of message batches it emitted to the sinkGAUGECOUNT
changefeed.checkpoint_progress
The earliest timestamp of any changefeed’s persisted checkpoint (values prior to this timestamp will never need to be re-emitted)GAUGETIMESTAMP_NS
changefeed.cloudstorage_buffered_bytes
The number of bytes buffered in cloudstorage sink files which have not been emitted yetGAUGECOUNT
changefeed.commit_latency
Event commit latency: a difference between event MVCC timestamp and the time it was acknowledged by the downstream sink. If the sink batches events, then the difference between the oldest event in the batch and acknowledgement is recorded. Excludes latency during backfill.HISTOGRAMNANOSECONDS
changefeed.emitted_batch_sizes
Size of batches emitted emitted by all feedsHISTOGRAMCOUNT
changefeed.emitted_bytes
Bytes emitted by all feedsCOUNTERBYTES
changefeed.emitted_messages
Messages emitted by all feedsCOUNTERCOUNT
changefeed.error_retries
Total retryable errors encountered by all changefeedsCOUNTERCOUNT
changefeed.filtered_messages
Messages filtered out by all feeds. This count does not include the number of messages that may be filtered due to the range constraints.COUNTERCOUNT
changefeed.flush_hist_nanos
Time spent flushing messages across all changefeedsHISTOGRAMNANOSECONDS
changefeed.flushed_bytes
Bytes emitted by all feeds; maybe different from changefeed.emitted_bytes when compression is enabledCOUNTERBYTES
changefeed.flushes
Total flushes across all feeds.COUNTERCOUNT
changefeed.internal_retry_message_count
Number of messages for which an attempt to retry them within an aggregator node was made.GAUGECOUNT
changefeed.kafka_throttling_hist_nanos
Time spent in throttling due to exceeding kafka quotaHISTOGRAMNANOSECONDS
changefeed.lagging_ranges
The number of ranges considered to be lagging behindGAUGECOUNT
changefeed.message_size_hist
Message size histogramHISTOGRAMBYTES
changefeed.parallel_io_in_flight_keys
The number of keys currently in-flight which may contend with batches pending to be emittedGAUGECOUNT
changefeed.parallel_io_pending_rows
Number of rows which are blocked from being sent due to conflicting in-flight keys.GAUGECOUNT
changefeed.parallel_io_queue_nanos
Time that outgoing requests to the sink spend waiting in a queue due to in-flight requests with conflicting keys.HISTOGRAMNANOSECONDS
changefeed.parallel_io_result_queue_nanos
Time that incoming results from the sink spend waiting in parallel io emitter before they are acknowledged by the changefeedHISTOGRAMNANOSECONDS
changefeed.running
Number of currently running changefeeds, including sinklessGAUGECOUNT
changefeed.schema_registry.registrations
Number of registration attempts with the schema registry.COUNTERCOUNT
changefeed.schema_registry.retry_count
Number of retries encountered when sending requests to the schema registry.COUNTERCOUNT
changefeed.sink_batch_hist_nanos
Time spent batched in the sink buffer before being flushed and acknowledged.HISTOGRAMNANOSECONDS
changefeed.sink_io_inflight
The number of keys currently inflight as IO requests being sent to the sinkGAUGECOUNT
changefeed.size_based_flushes
Total size based flushes across all feeds.COUNTERCOUNT

​
Clusters with logical data replication jobs

When child metrics is enabled and are created on the cluster, the logical_replication_*_by_label metrics are exported per LDR metric label. The label may have the values set using the label option. The cardinality increases with the number of LDR metric labels. For example, when you create two LDR jobs with the metrics labels ldr_job1 and ldr_job2, the metrics logical_replication_*_by_label export child metrics with a label for ldr_job1 and ldr_job2.
Note that the logical_replication_* metrics without the _by_label suffix may be inaccurate with multiple LDR jobs.
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescriptionTypeUnit
logical_replication.catchup_ranges_by_label
Source side ranges undergoing catch up scansGAUGECOUNT
logical_replication.events_dlqed_by_label
Row update events sent to DLQ by labelCOUNTERCOUNT
logical_replication.events_ingested_by_label
Events ingested by all replication jobs by labelCOUNTERCOUNT
logical_replication.replicated_time_by_label
Replicated time of the logical replication stream by labelGAUGESECONDS
logical_replication.scanning_ranges_by_label
Source side ranges undergoing an initial scanGAUGECOUNT

​
Clusters with row-level TTL jobs

When child metrics is enabled and are created on the cluster with the , the jobs.row_level_ttl.* metrics are exported per TTL job with ttl_label_metrics enabled with a label for relation. The value of the relation label may have the format: {database}_{schema}_{table}_{primary key}. The cardinality increases with the number of TTL jobs with ttl_label_metrics enabled. An aggregated metric is also included. For example:
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
jobs.row_level_ttl.delete_duration
Duration for delete requests during row level TTL.HISTOGRAMNANOSECONDS
jobs.row_level_ttl.num_active_spans
Number of active spans the TTL job is deleting from.GAUGECOUNT
jobs.row_level_ttl.rows_deleted
Number of rows deleted by the row level TTL job.COUNTERCOUNT
jobs.row_level_ttl.rows_selected
Number of rows selected for deletion by the row level TTL job.COUNTERCOUNT
jobs.row_level_ttl.select_duration
Duration for select requests during row level TTL.HISTOGRAMNANOSECONDS
jobs.row_level_ttl.span_total_duration
Duration for processing a span during row level TTL.HISTOGRAMNANOSECONDS
jobs.row_level_ttl.total_expired_rows
Approximate number of rows that have expired the TTL on the TTL table.GAUGECOUNT
jobs.row_level_ttl.total_rows
Approximate number of rows on the TTL table.GAUGECOUNT

​
Metrics of type histogram

When child metrics is enabled with changefeeds or row-level TTL jobs, be aware that metrics of type HISTOGRAM will increase cardinality quickly. For example, when you create two changefeeds with the metrics labels employees and office_dogs, the histogram metric changefeed_flush_hist_nanos exports child metrics for each bucket for each metrics label. In addition, the default scope will also be exported which includes changefeeds started without a metrics label. Therefore, in this example, changefeed_flush_hist_nanos exports child metrics for each bucket for the scope values: default, employees and office_dogs:

​
Enable database and application_name labels

This feature is in and subject to change. To share feedback and/or issues, contact Support.
The following cluster settings enable the database and application_name labels for certain metrics, along with their internal counterparts if they exist:
By default, these cluster settings are disabled. To enable them, use the statement. Because these labels use aggregate metrics, you must enable the server.child_metrics.enabled cluster setting to use them.
Toggling the sql.metrics.database_name.enabled and sql.metrics.application_name.enabled cluster settings clears existing metric values for current label combinations and reinitializes the affected metrics to reflect the new label configuration. When toggling the sql.metrics.database_name.enabled and sql.metrics.application_name.enabled cluster settings, only the values for existing metric label combinations will be cleared. Aggregated metric values for the affected metrics will not be cleared.
Child metrics (metrics with the database and application_name labels) are independent from the parent (aggregated metric). The child metrics are initialized when the cluster settings are enabled.For this reason, child COUNTER metrics may not always add up to the parent COUNTER metric. For an example, refer to Examples 1 through 6.For GAUGE metrics, values may be different and potentially unexpected depending on when a setting is enabled. For an example, refer to 7. GAUGE metric example.
These labels affect only the metrics emitted via . They are not visible in the . The system retains up to 5,000 recently used label combinations. Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
sql.crud_query.count
Number of SQL SELECT, INSERT, UPDATE, DELETE statements successfully executedCOUNTERCOUNT
sql.crud_query.count.internal
Number of SQL SELECT, INSERT, UPDATE, DELETE statements successfully executed (internal queries)COUNTERCOUNT
sql.delete.count
Number of SQL DELETE statements successfully executedCOUNTERCOUNT
sql.delete.count.internal
Number of SQL DELETE statements successfully executed (internal queries)COUNTERCOUNT
sql.distsql.contended_queries.count
Number of SQL queries that experienced contentionCOUNTERCOUNT
sql.distsql.cumulative_contention_nanos
Cumulative contention across all queries (in nanoseconds)COUNTERNANOSECONDS
sql.failure.count
Number of statements resulting in a planning or runtime errorCOUNTERCOUNT
sql.failure.count.internal
Number of statements resulting in a planning or runtime error (internal queries)COUNTERCOUNT
sql.full.scan.count
Number of full table or index scansCOUNTERCOUNT
sql.full.scan.count.internal
Number of full table or index scans (internal queries)COUNTERCOUNT
sql.insert.count
Number of SQL INSERT statements successfully executedCOUNTERCOUNT
sql.insert.count.internal
Number of SQL INSERT statements successfully executed (internal queries)COUNTERCOUNT
sql.select.count
Number of SQL SELECT statements successfully executedCOUNTERCOUNT
sql.select.count.internal
Number of SQL SELECT statements successfully executed (internal queries)COUNTERCOUNT
sql.service.latency
Latency of SQL request executionHISTOGRAMNANOSECONDS
sql.service.latency.internal
Latency of SQL request execution (internal queries)HISTOGRAMNANOSECONDS
sql.statements.active
Number of currently active user SQL statementsGAUGECOUNT
sql.statements.active.internal
Number of currently active user SQL statements (internal queries)GAUGECOUNT
sql.txn.begin.count
Number of SQL transaction BEGIN statements successfully executedCOUNTERCOUNT
sql.txn.begin.count.internal
Number of SQL transaction BEGIN statements successfully executed (internal queries)COUNTERCOUNT
sql.txn.commit.count
Number of SQL transaction COMMIT statements successfully executedCOUNTERCOUNT
sql.txn.commit.count.internal
Number of SQL transaction COMMIT statements successfully executed (internal queries)COUNTERCOUNT
sql.txn.latency
Latency of SQL transactionsHISTOGRAMNANOSECONDS
sql.txn.latency.internal
Latency of SQL transactions (internal queries)HISTOGRAMNANOSECONDS
sql.txn.rollback.count
Number of SQL transaction ROLLBACK statements successfully executedCOUNTERCOUNT
sql.txn.rollback.count.internal
Number of SQL transaction ROLLBACK statements successfully executed (internal queries)COUNTERCOUNT
sql.txns.open
Number of currently open user SQL transactionsGAUGECOUNT
sql.txns.open.internal
Number of currently open user SQL transactions (internal queries)GAUGECOUNT
sql.update.count
Number of SQL UPDATE statements successfully executedCOUNTERCOUNT
sql.update.count.internal
Number of SQL UPDATE statements successfully executed (internal queries)COUNTERCOUNT

​
Examples

This section demonstrates the impact of enabling and disabling the relevant cluster settings. Examples 1 through 6 show the effect on the COUNTER metric sql.select.count. During these examples, the was running on node 1. The aggregated metric consistently increases as the examples progress. Example 7 shows a possible effect on the GAUGE metric sql.txn.open.

​
1. All cluster settings disabled

The Prometheus export only gives the aggregated metric for the node.

​
2. Only child metrics enabled

The Prometheus export still only gives the aggregated metric for the node.

​
3. Child metrics and database_name label enabled

The aggregated metric and a child metric with only the database label are emitted.

​
4. Child metrics and application_name label enabled

The aggregated metric and a child metric with only the application_name label are emitted. Note that even though the aggregated metric has increased, the child metric with application_name label has a value less than the child metric with database label in the preceding example. This is because the labeled metrics have been reset, while the aggregated metric was not reset.

​
5. Child metrics and both database and application_name label enabled

The aggregated metric and a child metric with both database and application_name labels are emitted.

​
6. Aggregate metric disabled

The (default: true) reports an aggregate time series for applicable multi-dimensional metrics. When set to false, it stops reporting the aggregate time series, preventing double counting when querying those metrics.
No aggregated metric emitted. Only the child metric with both the database and application_name labels is emitted.

​
7. GAUGE metric example

Changes to cluster settings may take time to reinitialize affected metrics. As a result, some GAUGE metrics might briefly show unexpected values. GAUGE values for both aggregated and child metrics increase and decrease as transactions are opened and closed. This example uses the GAUGE metric sql.txn.open. Consider the following scenario:
TimeActionsql.txn.open aggregated metricsql.txn.open child metric
1Open a transaction. The value of the sql.txn.open aggregated metric is incremented.1-
2Enable sql.metrics.database_name.enabled and sql.metrics.application_name.enabled.
Re-initialize child metrics.		10
3Close a transaction.
The values of both the aggregated and child sql.txn.open metrics are decremented.		0-1
To avoid negative values in child metrics, use the Prometheus clamp_min function to set the metric to zero.

​
Enable detailed latency metrics

The labels the latency metric sql.exec.latency.detail with the . To estimate the cardinality of the set of all statement fingerprints, use the sql.query.unique.count metric. For most workloads, this metric ranges from dozens to hundreds. sql.stats.detailed_latency_metrics.enabled is disabled by default. To enable it, use the statement.
Be aware that the sql.exec.latency.detail metric is of type HISTOGRAM and will quickly increase the cardinality of exported metrics. For workloads with over a couple thousand statement fingerprints, exercise caution when enabling sql.stats.detailed_latency_metrics.enabled. For workloads with tens of thousands of distinct query fingerprints, leave this cluster setting set to false.
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescriptionTypeUnit
sql.exec.latency.detail
Latency of SQL statement execution, by statement fingerprintHISTOGRAMNANOSECONDS
For example: