Skip to main content
Multi-dimensional metrics are additional time series with extra labels. This page will help you understand the potential size of the Prometheus scrape payload for your workload when multi-dimensional metrics are enabled. The number of multi-dimensional metrics can significantly increase based on their associated labels, which increases cardinality. The export of multi-dimensional metrics can be enabled by two :

​
Enable child metrics

Child metrics are specific, detailed metrics that are usually related to a higher-level (parent or aggregate) metric. They often provide more granular or specific information about a particular aspect of the parent metric. The parent metrics and their potential child metrics are determined by the specific feature the cluster is using. The is disabled by default. To enable it, use the statement.
The (default: true) reports an aggregate time series for applicable multi-dimensional metrics. When set to false, it stops reporting the aggregate time series, preventing double counting when querying those metrics.

​
All clusters

An RPC (Remote Procedure Call) connection is a communication method used in distributed systems, like CockroachDB, to allow one program to request a service from a program located in another computer on a network without having to understand the network’s details. In the context of CockroachDB, RPC connections are used for inter-node communication. For instance, if Node 1 sends a request to Node 2, and Node 2 dials back (sends request back to Node 1), it ensures that communication is healthy in both directions. This is referred to as a “bidirectionally connected” and “heartbeating” RPC connection. When child metrics is enabled, for all clusters the rpc.connection.* metrics are exported per-peer with labels for remote_node_id, remote_addr, and class. The class label may have the following values: system, default, and raft. The cardinality increases with the number of nodes. An aggregated metric is also included. For example:
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
rpc.connection.avg_round_trip_latency
rpc.connection.failures
rpc.connection.healthy
rpc.connection.healthy_nanos
rpc.connection.heartbeats
rpc.connection.inactive
rpc.connection.unhealthy
rpc.connection.unhealthy_nanos

​
Secure clusters

When child metrics is enabled, for the security.certificate.expiration.client is exported per SQL user with a label for sql_user. The sql_user label may have the values of the cluster’s users who are logged into a node using client security certificates. The cardinality increases with the number of SQL users. An aggregated metric is also included, however since this is a sum of the child metric values which represent timestamps, it is not usable. For example:
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
security.certificate.expiration.client
Minimum expiration for client certificates, labeled by SQL user. 0 means no certificate or error.GAUGESECONDS

​
Virtual clusters

When child metrics is enabled, for the kv.tenant_rate_limit.* metrics and other kv-related metrics are exported per virtual cluster with a label for tenant_id. The tenant_id label may have the values: system or the id of the virtual cluster. The cardinality increases with the number of virtual clusters. An aggregated metric is also included.
With virtual clusters, while the tenant_id label on kv metrics is only exported when child metrics is enabled, is exported whether child metrics is enabled or disabled.
For example:
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
abortspanbytes
gcbytesage
intentage
intentbytes
intentcount
keybytes
keycount
kv.tenant_rate_limit.current_blocked
kv.tenant_rate_limit.num_tenants
kv.tenant_rate_limit.read_batches_admitted
kv.tenant_rate_limit.read_bytes_admitted
kv.tenant_rate_limit.read_requests_admitted
kv.tenant_rate_limit.write_batches_admitted
kv.tenant_rate_limit.write_bytes_admitted
kv.tenant_rate_limit.write_requests_admitted
livebytes
livecount
lockbytes
lockcount
rangekeybytes
rangekeycount
rangevalbytes
rangevalcount
sysbytes
syscount
totalbytes
valbytes
valcount

​
Clusters with changefeeds

When child metrics is enabled and are created on the cluster, the changefeed.* metrics are exported per changefeed metric label with a label for scope. The scope label may have the values set using the metrics_label option. The cardinality increases with the number of changefeed metric labels. An aggregated metric is also included. For example, when you create two changefeeds with the metrics labels employees and office_dogs, the counter metric changefeed_error_retries exports child metrics with a scope for employees and office_dogs. In addition, the default scope will also be exported which includes changefeeds started without a metrics label.
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
changefeed.admit_latency
changefeed.aggregator_progress
changefeed.backfill_count
changefeed.backfill_pending_ranges
changefeed.batch_reduction_count
changefeed.checkpoint_progress
changefeed.cloudstorage_buffered_bytes
changefeed.commit_latency
changefeed.emitted_batch_sizes
changefeed.emitted_bytes
changefeed.emitted_messages
changefeed.error_retries
changefeed.filtered_messages
changefeed.flush_hist_nanos
changefeed.flushed_bytes
changefeed.flushes
changefeed.internal_retry_message_count
changefeed.kafka_throttling_hist_nanos
changefeed.lagging_ranges
changefeed.message_size_hist
changefeed.parallel_io_in_flight_keys
changefeed.parallel_io_pending_rows
changefeed.parallel_io_queue_nanos
changefeed.parallel_io_result_queue_nanos
changefeed.running
changefeed.schema_registry.registrations
changefeed.schema_registry.retry_count
changefeed.sink_batch_hist_nanos
changefeed.sink_io_inflight
changefeed.size_based_flushes

​
Clusters with logical data replication jobs

When child metrics is enabled and are created on the cluster, the logical_replication_*_by_label metrics are exported per LDR metric label. The label may have the values set using the label option. The cardinality increases with the number of LDR metric labels. For example, when you create two LDR jobs with the metrics labels ldr_job1 and ldr_job2, the metrics logical_replication_*_by_label export child metrics with a label for ldr_job1 and ldr_job2.
Note that the logical_replication_* metrics without the _by_label suffix may be inaccurate with multiple LDR jobs.
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescriptionTypeUnit
logical_replication.catchup_ranges_by_label
logical_replication.events_dlqed_by_label
logical_replication.events_ingested_by_label
logical_replication.replicated_time_by_label
logical_replication.scanning_ranges_by_label

​
Clusters with row-level TTL jobs

When child metrics is enabled and are created on the cluster with the , the jobs.row_level_ttl.* metrics are exported per TTL job with ttl_label_metrics enabled with a label for relation. The value of the relation label may have the format: {database}_{schema}_{table}_{primary key}. The cardinality increases with the number of TTL jobs with ttl_label_metrics enabled. An aggregated metric is also included. For example:
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
jobs.row_level_ttl.delete_duration
jobs.row_level_ttl.num_active_spans
jobs.row_level_ttl.rows_deleted
jobs.row_level_ttl.rows_selected
jobs.row_level_ttl.select_duration
jobs.row_level_ttl.span_total_duration
jobs.row_level_ttl.total_expired_rows
jobs.row_level_ttl.total_rows

​
Metrics of type histogram

When child metrics is enabled with changefeeds or row-level TTL jobs, be aware that metrics of type HISTOGRAM will increase cardinality quickly. For example, when you create two changefeeds with the metrics labels employees and office_dogs, the histogram metric changefeed_flush_hist_nanos exports child metrics for each bucket for each metrics label. In addition, the default scope will also be exported which includes changefeeds started without a metrics label. Therefore, in this example, changefeed_flush_hist_nanos exports child metrics for each bucket for the scope values: default, employees and office_dogs:

​
Enable database and application_name labels

This feature is in and subject to change. To share feedback and/or issues, contact Support.
The following cluster settings enable the database and application_name labels for certain metrics, along with their internal counterparts if they exist:
By default, these cluster settings are disabled. To enable them, use the statement. Because these labels use aggregate metrics, you must enable the server.child_metrics.enabled cluster setting to use them.
Toggling the sql.metrics.database_name.enabled and sql.metrics.application_name.enabled cluster settings clears existing metric values for current label combinations and reinitializes the affected metrics to reflect the new label configuration. When toggling the sql.metrics.database_name.enabled and sql.metrics.application_name.enabled cluster settings, only the values for existing metric label combinations will be cleared. Aggregated metric values for the affected metrics will not be cleared.
Child metrics (metrics with the database and application_name labels) are independent from the parent (aggregated metric). The child metrics are initialized when the cluster settings are enabled.For this reason, child COUNTER metrics may not always add up to the parent COUNTER metric. For an example, refer to Examples 1 through 6.For GAUGE metrics, values may be different and potentially unexpected depending on when a setting is enabled. For an example, refer to 7. GAUGE metric example.
These labels affect only the metrics emitted via . They are not visible in the . The system retains up to 5,000 recently used label combinations. Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
sql.crud_query.count
sql.crud_query.count.internal
sql.delete.count
sql.delete.count.internal
sql.distsql.contended_queries.count
sql.distsql.cumulative_contention_nanos
sql.failure.count
sql.failure.count.internal
sql.full.scan.count
sql.full.scan.count.internal
sql.insert.count
sql.insert.count.internal
sql.select.count
sql.select.count.internal
sql.service.latency
sql.service.latency.internal
sql.statements.active
sql.statements.active.internal
sql.txn.begin.count
sql.txn.begin.count.internal
sql.txn.commit.count
sql.txn.commit.count.internal
sql.txn.latency
sql.txn.latency.internal
sql.txn.rollback.count
sql.txn.rollback.count.internal
sql.txns.open
sql.txns.open.internal
sql.update.count
sql.update.count.internal

​
Examples

This section demonstrates the impact of enabling and disabling the relevant cluster settings. Examples 1 through 6 show the effect on the COUNTER metric sql.select.count. During these examples, the was running on node 1. The aggregated metric consistently increases as the examples progress. Example 7 shows a possible effect on the GAUGE metric sql.txn.open.

​
1. All cluster settings disabled

The Prometheus export only gives the aggregated metric for the node.

​
2. Only child metrics enabled

The Prometheus export still only gives the aggregated metric for the node.

​
3. Child metrics and database_name label enabled

The aggregated metric and a child metric with only the database label are emitted.

​
4. Child metrics and application_name label enabled

The aggregated metric and a child metric with only the application_name label are emitted. Note that even though the aggregated metric has increased, the child metric with application_name label has a value less than the child metric with database label in the preceding example. This is because the labeled metrics have been reset, while the aggregated metric was not reset.

​
5. Child metrics and both database and application_name label enabled

The aggregated metric and a child metric with both database and application_name labels are emitted.

​
6. Aggregate metric disabled

The (default: true) reports an aggregate time series for applicable multi-dimensional metrics. When set to false, it stops reporting the aggregate time series, preventing double counting when querying those metrics.
No aggregated metric emitted. Only the child metric with both the database and application_name labels is emitted.

​
7. GAUGE metric example

Changes to cluster settings may take time to reinitialize affected metrics. As a result, some GAUGE metrics might briefly show unexpected values. GAUGE values for both aggregated and child metrics increase and decrease as transactions are opened and closed. This example uses the GAUGE metric sql.txn.open. Consider the following scenario:
TimeActionsql.txn.open aggregated metricsql.txn.open child metric
1Open a transaction. The value of the sql.txn.open aggregated metric is incremented.1-
2Enable sql.metrics.database_name.enabled and sql.metrics.application_name.enabled.
Re-initialize child metrics.		10
3Close a transaction.
The values of both the aggregated and child sql.txn.open metrics are decremented.		0-1
To avoid negative values in child metrics, use the Prometheus clamp_min function to set the metric to zero.

​
Enable detailed latency metrics

The labels the latency metric sql.exec.latency.detail with the . To estimate the cardinality of the set of all statement fingerprints, use the sql.query.unique.count metric. For most workloads, this metric ranges from dozens to hundreds. sql.stats.detailed_latency_metrics.enabled is disabled by default. To enable it, use the statement.
Be aware that the sql.exec.latency.detail metric is of type HISTOGRAM and will quickly increase the cardinality of exported metrics. For workloads with over a couple thousand statement fingerprints, exercise caution when enabling sql.stats.detailed_latency_metrics.enabled. For workloads with tens of thousands of distinct query fingerprints, leave this cluster setting set to false.
Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescriptionTypeUnit
sql.exec.latency.detail
For example: