> ## Documentation Index
> Fetch the complete documentation index at: https://cockroachlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-dimensional Metrics

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

Multi-dimensional metrics are additional <InternalLink path="prometheus-endpoint">Prometheus</InternalLink> time series with extra labels. This page will help you understand the potential size of the Prometheus scrape payload for your workload when multi-dimensional metrics are enabled. The number of multi-dimensional metrics can significantly increase based on their associated labels, which increases cardinality.

The export of multi-dimensional metrics can be enabled by two <InternalLink path="cluster-settings">cluster settings</InternalLink>:

* [`server.child_metrics.enabled`](#enable-child-metrics)
* [`sql.metrics.database_name.enabled`](#enable-database-and-application_name-labels)
* [`sql.metrics.application_name.enabled`](#enable-database-and-application_name-labels)
* [`sql.stats.detailed_latency_metrics.enabled`](#enable-detailed-latency-metrics)

## Enable child metrics

Child metrics are specific, detailed metrics that are usually related to a higher-level (parent or aggregate) metric. They often provide more granular or specific information about a particular aspect of the parent metric. The parent metrics and their potential child metrics are determined by the specific feature the cluster is using.

The <InternalLink path="cluster-settings#setting-server-child-metrics-enabled">cluster setting `server.child_metrics.enabled`</InternalLink> is disabled by default. To enable it, use the <InternalLink path="set-cluster-setting">`SET CLUSTER SETTING`</InternalLink> statement.

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING server.child_metrics.enabled = true;
```

The <InternalLink path="cluster-settings#setting-server-child-metrics-include-aggregate-enabled">cluster setting `server.child_metrics.include_aggregate.enabled`</InternalLink> (default: `true`) reports an aggregate time series for applicable multi-dimensional metrics. When set to `false`, it stops reporting the aggregate time series, preventing double counting when querying those metrics.

### All clusters

An RPC (Remote Procedure Call) connection is a communication method used in distributed systems, like CockroachDB, to allow one program to request a service from a program located in another computer on a network without having to understand the network's details. In the context of CockroachDB, RPC connections are used for inter-node communication. For instance, if Node 1 sends a request to Node 2, and Node 2 dials back (sends request back to Node 1), it ensures that communication is healthy in both directions. This is referred to as a "bidirectionally connected" and "heartbeating" RPC connection.

When child metrics is enabled, for all clusters the `rpc.connection.*` metrics are exported per-peer with labels for `remote_node_id`, `remote_addr`, and `class`. The `class` label may have the following values: `system`, `default`, and `raft`. The cardinality increases with the number of nodes. An aggregated metric is also included.

For example:

```
# HELP rpc_connection_healthy Gauge of current connections in a healthy state (i.e. bidirectionally connected and heartbeating)
# TYPE rpc_connection_healthy gauge
rpc_connection_healthy{node_id="1"} 10
rpc_connection_healthy{node_id="1",remote_node_id="0",remote_addr="127.0.0.1:29004",class="system"} 1
rpc_connection_healthy{node_id="1",remote_node_id="0",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29000",class="system"} 0
rpc_connection_healthy{node_id="1",remote_node_id="0",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29002",class="system"} 1
rpc_connection_healthy{node_id="1",remote_node_id="1",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29004",class="default"} 1
rpc_connection_healthy{node_id="1",remote_node_id="1",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29004",class="system"} 1
rpc_connection_healthy{node_id="1",remote_node_id="2",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29002",class="default"} 1
rpc_connection_healthy{node_id="1",remote_node_id="2",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29002",class="raft"} 1
rpc_connection_healthy{node_id="1",remote_node_id="2",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29002",class="system"} 1
rpc_connection_healthy{node_id="1",remote_node_id="3",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29000",class="default"} 1
rpc_connection_healthy{node_id="1",remote_node_id="3",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29000",class="raft"} 1
rpc_connection_healthy{node_id="1",remote_node_id="3",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29000",class="system"} 1
```

Following is a list of the metrics that have multi-dimensional metrics:

<table className="crdb-fixed-table"><thead><tr><th style={{width: "30%"}}>CockroachDB Metric Name</th><th style={{width: "44%"}}>Description When Aggregated</th><th style={{width: "13%"}}>Type</th><th style={{width: "13%"}}>Unit</th></tr></thead><tbody> <tr><td><div id="rpc.connection.avg_round_trip_latency" class="anchored"><code>rpc.connection.avg\_round\_trip\_latency</code></div></td><td>Sum of exponentially weighted moving average of round-trip latencies, as measured through a gRPC RPC. Since this metric is based on gRPC RPCs, it is affected by application-level processing delays and CPU overload effects. See rpc.connection.tcp\_rtt for a metric that is obtained from the kernel's TCP stack. Dividing this Gauge by rpc.connection.healthy gives an approximation of average latency, but the top-level round-trip-latency histogram is more useful. Instead, users should consult the label families of this metric if they are available (which requires prometheus and the cluster setting 'server.child\_metrics.enabled'); these provide per-peer moving averages. This metric does not track failed connection. A failed connection's contribution is reset to zero. </td><td>GAUGE</td><td>NANOSECONDS</td></tr> <tr><td><div id="rpc.connection.failures" class="anchored"><code>rpc.connection.failures</code></div></td><td>Counter of failed connections. This includes both the event in which a healthy connection terminates as well as unsuccessful reconnection attempts. Connections that are terminated as part of local node shutdown are excluded. Decommissioned peers are excluded. </td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="rpc.connection.healthy" class="anchored"><code>rpc.connection.healthy</code></div></td><td>Gauge of current connections in a healthy state (i.e. bidirectionally connected and heartbeating)</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="rpc.connection.healthy_nanos" class="anchored"><code>rpc.connection.healthy\_nanos</code></div></td><td>Gauge of nanoseconds of healthy connection time On the prometheus endpoint scraped with the cluster setting 'server.child\_metrics.enabled' set, the constituent parts of this metric are available on a per-peer basis and one can read off for how long a given peer has been connected</td><td>GAUGE</td><td>NANOSECONDS</td></tr> <tr><td><div id="rpc.connection.heartbeats" class="anchored"><code>rpc.connection.heartbeats</code></div></td><td>Counter of successful heartbeats.</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="rpc.connection.inactive" class="anchored"><code>rpc.connection.inactive</code></div></td><td>Gauge of current connections in an inactive state and pending deletion; these are not healthy but are not tracked as unhealthy either because there is reason to believe that the connection is no longer relevant,for example if the node has since been seen under a new address</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="rpc.connection.unhealthy" class="anchored"><code>rpc.connection.unhealthy</code></div></td><td>Gauge of current connections in an unhealthy state (not bidirectionally connected or heartbeating)</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="rpc.connection.unhealthy_nanos" class="anchored"><code>rpc.connection.unhealthy\_nanos</code></div></td><td>Gauge of nanoseconds of unhealthy connection time. On the prometheus endpoint scraped with the cluster setting 'server.child\_metrics.enabled' set, the constituent parts of this metric are available on a per-peer basis and one can read off for how long a given peer has been unreachable</td><td>GAUGE</td><td>NANOSECONDS</td></tr> </tbody></table>

### Secure clusters

When child metrics is enabled, for <InternalLink path="secure-a-cluster">secure clusters</InternalLink> the `security.certificate.expiration.client` is exported per SQL user with a label for `sql_user`. The `sql_user` label may have the values of the cluster's users who are logged into a node using client security certificates. The cardinality increases with the number of SQL users. An aggregated metric is also included, however since this is a sum of the child metric values which represent timestamps, it is not usable.

For example:

```
# HELP security_certificate_expiration_client Minimum expiration for client certificates, labeled by SQL user. 0 means no certificate or error. 
# TYPE security_certificate_expiration_client gauge
security_certificate_expiration_client{node_id="1"} 3.756110141e+09
security_certificate_expiration_client{node_id="1",sql_user="florence"} 1.878055127e+09
security_certificate_expiration_client{node_id="1",sql_user="root"} 1.878055014e+09
```

Following is a list of the metrics that have multi-dimensional metrics:

<table className="crdb-fixed-table"><thead><tr><th style={{width: "30%"}}>CockroachDB Metric Name</th><th style={{width: "44%"}}>Description When Aggregated</th><th style={{width: "13%"}}>Type</th><th style={{width: "13%"}}>Unit</th></tr></thead><tbody> <tr><td><div id="security.certificate.expiration.client" class="anchored"><code>security.certificate.expiration.client</code></div></td><td>Minimum expiration for client certificates, labeled by SQL user. 0 means no certificate or error.</td><td>GAUGE</td><td>SECONDS</td></tr> </tbody></table>

### Virtual clusters

When child metrics is enabled, for <InternalLink path="cluster-virtualization-overview">virtual clusters</InternalLink> the `kv.tenant_rate_limit.*` metrics and other kv-related metrics are exported per virtual cluster with a label for `tenant_id`. The `tenant_id` label may have the values: `system` or the id of the virtual cluster. The cardinality increases with the number of virtual clusters. An aggregated metric is also included.

<Note>
  With virtual clusters, while the `tenant_id` label on kv metrics is only exported when child metrics is enabled, <InternalLink path="work-with-virtual-clusters#work-with-metrics-with-cluster-virtualization-enabled">the `tenant` label on SQL metrics</InternalLink> is exported whether child metrics is enabled or disabled.
</Note>

For example:

```
# HELP kv_tenant_rate_limit_read_bytes_admitted Number of read bytes admitted by the rate limiter
# TYPE kv_tenant_rate_limit_read_bytes_admitted counter
kv_tenant_rate_limit_read_bytes_admitted{store="1",node_id="1"} 41
kv_tenant_rate_limit_read_bytes_admitted{store="1",node_id="1",tenant_id="3"} 41
kv_tenant_rate_limit_read_bytes_admitted{store="1",node_id="1",tenant_id="4"} 0
kv_tenant_rate_limit_read_bytes_admitted{store="1",node_id="1",tenant_id="system"} 0

# HELP sysbytes Number of bytes in system KV pairs
# TYPE sysbytes gauge
sysbytes{store="1",node_id="1"} 41979
sysbytes{store="1",node_id="1",tenant_id="3"} 278
sysbytes{store="1",node_id="1",tenant_id="4"} 225
sysbytes{store="1",node_id="1",tenant_id="system"} 41476
```

Following is a list of the metrics that have multi-dimensional metrics:

<table className="crdb-fixed-table"><thead><tr><th style={{width: "30%"}}>CockroachDB Metric Name</th><th style={{width: "44%"}}>Description When Aggregated</th><th style={{width: "13%"}}>Type</th><th style={{width: "13%"}}>Unit</th></tr></thead><tbody> <tr><td><div id="abortspanbytes" class="anchored"><code>abortspanbytes</code></div></td><td>Number of bytes in the abort span</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="gcbytesage" class="anchored"><code>gcbytesage</code></div></td><td>Cumulative age of non-live data</td><td>GAUGE</td><td>SECONDS</td></tr> <tr><td><div id="intentage" class="anchored"><code>intentage</code></div></td><td>Cumulative age of locks</td><td>GAUGE</td><td>SECONDS</td></tr> <tr><td><div id="intentbytes" class="anchored"><code>intentbytes</code></div></td><td>Number of bytes in intent KV pairs</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="intentcount" class="anchored"><code>intentcount</code></div></td><td>Count of intent keys</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="keybytes" class="anchored"><code>keybytes</code></div></td><td>Number of bytes taken up by keys</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="keycount" class="anchored"><code>keycount</code></div></td><td>Count of all keys</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="kv.tenant_rate_limit.current_blocked" class="anchored"><code>kv.tenant\_rate\_limit.current\_blocked</code></div></td><td>Number of requests currently blocked by the rate limiter</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="kv.tenant_rate_limit.num_tenants" class="anchored"><code>kv.tenant\_rate\_limit.num\_tenants</code></div></td><td>Number of tenants currently being tracked</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="kv.tenant_rate_limit.read_batches_admitted" class="anchored"><code>kv.tenant\_rate\_limit.read\_batches\_admitted</code></div></td><td>Number of read batches admitted by the rate limiter</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="kv.tenant_rate_limit.read_bytes_admitted" class="anchored"><code>kv.tenant\_rate\_limit.read\_bytes\_admitted</code></div></td><td>Number of read bytes admitted by the rate limiter</td><td>COUNTER</td><td>BYTES</td></tr> <tr><td><div id="kv.tenant_rate_limit.read_requests_admitted" class="anchored"><code>kv.tenant\_rate\_limit.read\_requests\_admitted</code></div></td><td>Number of read requests admitted by the rate limiter</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="kv.tenant_rate_limit.write_batches_admitted" class="anchored"><code>kv.tenant\_rate\_limit.write\_batches\_admitted</code></div></td><td>Number of write batches admitted by the rate limiter</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="kv.tenant_rate_limit.write_bytes_admitted" class="anchored"><code>kv.tenant\_rate\_limit.write\_bytes\_admitted</code></div></td><td>Number of write bytes admitted by the rate limiter</td><td>COUNTER</td><td>BYTES</td></tr> <tr><td><div id="kv.tenant_rate_limit.write_requests_admitted" class="anchored"><code>kv.tenant\_rate\_limit.write\_requests\_admitted</code></div></td><td>Number of write requests admitted by the rate limiter</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="livebytes" class="anchored"><code>livebytes</code></div></td><td>Number of bytes of live data (keys plus values)</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="livecount" class="anchored"><code>livecount</code></div></td><td>Count of live keys</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="lockbytes" class="anchored"><code>lockbytes</code></div></td><td>Number of bytes taken up by replicated lock key-values (shared and exclusive strength, not intent strength)</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="lockcount" class="anchored"><code>lockcount</code></div></td><td>Count of replicated locks (shared, exclusive, and intent strength)</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="rangekeybytes" class="anchored"><code>rangekeybytes</code></div></td><td>Number of bytes taken up by range keys (e.g. MVCC range tombstones)</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="rangekeycount" class="anchored"><code>rangekeycount</code></div></td><td>Count of all range keys (e.g. MVCC range tombstones)</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="rangevalbytes" class="anchored"><code>rangevalbytes</code></div></td><td>Number of bytes taken up by range key values (e.g. MVCC range tombstones)</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="rangevalcount" class="anchored"><code>rangevalcount</code></div></td><td>Count of all range key values (e.g. MVCC range tombstones)</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="sysbytes" class="anchored"><code>sysbytes</code></div></td><td>Number of bytes in system KV pairs</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="syscount" class="anchored"><code>syscount</code></div></td><td>Count of system KV pairs</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="totalbytes" class="anchored"><code>totalbytes</code></div></td><td>Total number of bytes taken up by keys and values including non-live data</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="valbytes" class="anchored"><code>valbytes</code></div></td><td>Number of bytes taken up by values</td><td>GAUGE</td><td>BYTES</td></tr> <tr><td><div id="valcount" class="anchored"><code>valcount</code></div></td><td>Count of all values</td><td>GAUGE</td><td>COUNT</td></tr> </tbody></table>

### Clusters with changefeeds

When child metrics is enabled and <InternalLink path="monitor-and-debug-changefeeds#using-changefeed-metrics-labels">changefeeds with metrics labels</InternalLink> are created on the cluster, the `changefeed.*` metrics are exported per changefeed metric label with a label for `scope`. The `scope` label may have the values set using the `metrics_label` option. The cardinality increases with the number of changefeed metric labels. An aggregated metric is also included.

For example, when you create two changefeeds with the metrics labels `employees` and `office_dogs`, the counter metric `changefeed_error_retries` exports child metrics with a `scope` for `employees` and `office_dogs`. In addition, the `default` scope will also be exported which includes changefeeds started without a metrics label.

```
 HELP changefeed_error_retries Total retryable errors encountered by all changefeeds
# TYPE changefeed_error_retries counter
changefeed_error_retries{node_id="1"} 0
changefeed_error_retries{node_id="1",scope="default"} 0
changefeed_error_retries{node_id="1",scope="employees"} 0
changefeed_error_retries{node_id="1",scope="office_dogs"} 0
```

Following is a list of the metrics that have multi-dimensional metrics:

<table className="crdb-fixed-table"><thead><tr><th style={{width: "30%"}}>CockroachDB Metric Name</th><th style={{width: "44%"}}>Description When Aggregated</th><th style={{width: "13%"}}>Type</th><th style={{width: "13%"}}>Unit</th></tr></thead><tbody> <tr><td><div id="changefeed.admit_latency" class="anchored"><code>changefeed.admit\_latency</code></div></td><td>Event admission latency: a difference between event MVCC timestamp and the time it was admitted into changefeed pipeline; Note: this metric includes the time spent waiting until event can be processed due to backpressure or time spent resolving schema descriptors. Also note, this metric excludes latency during backfill</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="changefeed.aggregator_progress" class="anchored"><code>changefeed.aggregator\_progress</code></div></td><td>The earliest timestamp up to which any aggregator is guaranteed to have emitted all values for</td><td>GAUGE</td><td>TIMESTAMP\_NS</td></tr> <tr><td><div id="changefeed.backfill_count" class="anchored"><code>changefeed.backfill\_count</code></div></td><td>Number of changefeeds currently executing backfill</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="changefeed.backfill_pending_ranges" class="anchored"><code>changefeed.backfill\_pending\_ranges</code></div></td><td>Number of ranges in an ongoing backfill that are yet to be fully emitted</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="changefeed.batch_reduction_count" class="anchored"><code>changefeed.batch\_reduction\_count</code></div></td><td>Number of times a changefeed aggregator node attempted to reduce the size of message batches it emitted to the sink</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="changefeed.checkpoint_progress" class="anchored"><code>changefeed.checkpoint\_progress</code></div></td><td>The earliest timestamp of any changefeed's persisted checkpoint (values prior to this timestamp will never need to be re-emitted)</td><td>GAUGE</td><td>TIMESTAMP\_NS</td></tr> <tr><td><div id="changefeed.cloudstorage_buffered_bytes" class="anchored"><code>changefeed.cloudstorage\_buffered\_bytes</code></div></td><td>The number of bytes buffered in cloudstorage sink files which have not been emitted yet</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="changefeed.commit_latency" class="anchored"><code>changefeed.commit\_latency</code></div></td><td>Event commit latency: a difference between event MVCC timestamp and the time it was acknowledged by the downstream sink.  If the sink batches events,  then the difference between the oldest event in the batch and acknowledgement is recorded; Excludes latency during backfill</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="changefeed.emitted_batch_sizes" class="anchored"><code>changefeed.emitted\_batch\_sizes</code></div></td><td>Size of batches emitted emitted by all feeds</td><td>HISTOGRAM</td><td>COUNT</td></tr> <tr><td><div id="changefeed.emitted_bytes" class="anchored"><code>changefeed.emitted\_bytes</code></div></td><td>Bytes emitted by all feeds</td><td>COUNTER</td><td>BYTES</td></tr> <tr><td><div id="changefeed.emitted_messages" class="anchored"><code>changefeed.emitted\_messages</code></div></td><td>Messages emitted by all feeds</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="changefeed.error_retries" class="anchored"><code>changefeed.error\_retries</code></div></td><td>Total retryable errors encountered by all changefeeds</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="changefeed.filtered_messages" class="anchored"><code>changefeed.filtered\_messages</code></div></td><td>Messages filtered out by all feeds. This count does not include the number of messages that may be filtered due to the range constraints.</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="changefeed.flush_hist_nanos" class="anchored"><code>changefeed.flush\_hist\_nanos</code></div></td><td>Time spent flushing messages across all changefeeds</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="changefeed.flushed_bytes" class="anchored"><code>changefeed.flushed\_bytes</code></div></td><td>Bytes emitted by all feeds; maybe different from changefeed.emitted\_bytes when compression is enabled</td><td>COUNTER</td><td>BYTES</td></tr> <tr><td><div id="changefeed.flushes" class="anchored"><code>changefeed.flushes</code></div></td><td>Total flushes across all feeds</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="changefeed.internal_retry_message_count" class="anchored"><code>changefeed.internal\_retry\_message\_count</code></div></td><td>Number of messages for which an attempt to retry them within an aggregator node was made</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="changefeed.kafka_throttling_hist_nanos" class="anchored"><code>changefeed.kafka\_throttling\_hist\_nanos</code></div></td><td>Time spent in throttling due to exceeding kafka quota</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="changefeed.lagging_ranges" class="anchored"><code>changefeed.lagging\_ranges</code></div></td><td>The number of ranges considered to be lagging behind</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="changefeed.message_size_hist" class="anchored"><code>changefeed.message\_size\_hist</code></div></td><td>Message size histogram</td><td>HISTOGRAM</td><td>BYTES</td></tr> <tr><td><div id="changefeed.parallel_io_in_flight_keys" class="anchored"><code>changefeed.parallel\_io\_in\_flight\_keys</code></div></td><td>The number of keys currently in-flight which may contend with batches pending to be emitted</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="changefeed.parallel_io_pending_rows" class="anchored"><code>changefeed.parallel\_io\_pending\_rows</code></div></td><td>Number of rows which are blocked from being sent due to conflicting in-flight keys</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="changefeed.parallel_io_queue_nanos" class="anchored"><code>changefeed.parallel\_io\_queue\_nanos</code></div></td><td>Time that outgoing requests to the sink spend waiting in a queue due to in-flight requests with conflicting keys</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="changefeed.parallel_io_result_queue_nanos" class="anchored"><code>changefeed.parallel\_io\_result\_queue\_nanos</code></div></td><td>Time that incoming results from the sink spend waiting in parallel io emitter before they are acknowledged by the changefeed</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="changefeed.running" class="anchored"><code>changefeed.running</code></div></td><td>Number of currently running changefeeds, including sinkless</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="changefeed.schema_registry.registrations" class="anchored"><code>changefeed.schema\_registry.registrations</code></div></td><td>Number of registration attempts with the schema registry</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="changefeed.schema_registry.retry_count" class="anchored"><code>changefeed.schema\_registry.retry\_count</code></div></td><td>Number of retries encountered when sending requests to the schema registry</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="changefeed.sink_batch_hist_nanos" class="anchored"><code>changefeed.sink\_batch\_hist\_nanos</code></div></td><td>Time spent batched in the sink buffer before being flushed and acknowledged</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="changefeed.sink_io_inflight" class="anchored"><code>changefeed.sink\_io\_inflight</code></div></td><td>The number of keys currently inflight as IO requests being sent to the sink</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="changefeed.size_based_flushes" class="anchored"><code>changefeed.size\_based\_flushes</code></div></td><td>Total size based flushes across all feeds</td><td>COUNTER</td><td>COUNT</td></tr> </tbody></table>

### Clusters with logical data replication jobs

When child metrics is enabled and <InternalLink path="logical-data-replication-monitoring#metrics-labels">logical data replication (LDR) jobs with metrics labels</InternalLink> are created on the cluster, the `logical_replication_*_by_label` metrics are exported per LDR metric label. The `label` may have the values set using the `label` option. The cardinality increases with the number of LDR metric labels.

For example, when you create two LDR jobs with the metrics labels `ldr_job1` and `ldr_job2`, the metrics `logical_replication_*_by_label` export child metrics with a `label` for `ldr_job1` and `ldr_job2`.

```
# HELP logical_replication_replicated_time_by_label Replicated time of the logical replication stream by label
# TYPE logical_replication_replicated_time_by_label gauge
logical_replication_replicated_time_by_label{label="ldr_job2",node_id="2"} 1.73411035e+09
logical_replication_replicated_time_by_label{label="ldr_job1",node_id="2"} 1.73411035e+09
# HELP logical_replication_catchup_ranges_by_label Source side ranges undergoing catch up scans
# TYPE logical_replication_catchup_ranges_by_label gauge
logical_replication_catchup_ranges_by_label{label="ldr_job1",node_id="2"} 0
logical_replication_catchup_ranges_by_label{label="ldr_job2",node_id="2"} 0
# HELP logical_replication_scanning_ranges_by_label Source side ranges undergoing an initial scan
# TYPE logical_replication_scanning_ranges_by_label gauge
logical_replication_scanning_ranges_by_label{label="ldr_job1",node_id="2"} 0
logical_replication_scanning_ranges_by_label{label="ldr_job2",node_id="2"} 0
```

Note that the `logical_replication_*` metrics without the `_by_label` suffix may be `inaccurate with multiple LDR jobs`.

```
# HELP logical_replication_catchup_ranges Source side ranges undergoing catch up scans (inaccurate with multiple LDR jobs)
# TYPE logical_replication_catchup_ranges gauge
logical_replication_catchup_ranges{node_id="2"} 0
# HELP logical_replication_scanning_ranges Source side ranges undergoing an initial scan (inaccurate with multiple LDR jobs)
# TYPE logical_replication_scanning_ranges gauge
logical_replication_scanning_ranges{node_id="2"} 0
```

Following is a list of the metrics that have multi-dimensional metrics:

<table className="crdb-fixed-table"><thead><tr><th style={{width: "30%"}}>CockroachDB Metric Name</th><th style={{width: "44%"}}>Description</th><th style={{width: "13%"}}>Type</th><th style={{width: "13%"}}>Unit</th></tr></thead><tbody> <tr><td><div id="logical_replication.catchup_ranges_by_label" class="anchored"><code>logical\_replication.catchup\_ranges\_by\_label</code></div></td><td>Source side ranges undergoing catch up scans</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="logical_replication.events_dlqed_by_label" class="anchored"><code>logical\_replication.events\_dlqed\_by\_label</code></div></td><td>Row update events sent to DLQ by label</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="logical_replication.events_ingested_by_label" class="anchored"><code>logical\_replication.events\_ingested\_by\_label</code></div></td><td>Events ingested by all replication jobs by label</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="logical_replication.replicated_time_by_label" class="anchored"><code>logical\_replication.replicated\_time\_by\_label</code></div></td><td>Replicated time of the logical replication stream by label</td><td>GAUGE</td><td>SECONDS</td></tr> <tr><td><div id="logical_replication.scanning_ranges_by_label" class="anchored"><code>logical\_replication.scanning\_ranges\_by\_label</code></div></td><td>Source side ranges undergoing an initial scan</td><td>GAUGE</td><td>COUNT</td></tr> </tbody></table>

### Clusters with row-level TTL jobs

When child metrics is enabled and <InternalLink path="row-level-ttl">row-level TTL jobs</InternalLink> are created on the cluster with the <InternalLink path="row-level-ttl#ttl-metrics">`ttl_label_metrics` storage parameter enabled</InternalLink>, the `jobs.row_level_ttl.*` metrics are exported per TTL job with `ttl_label_metrics` enabled with a label for `relation`. The value of the `relation` label may have the format: `{database}_{schema}_{table}_{primary key}`. The cardinality increases with the number of TTL jobs with `ttl_label_metrics` enabled. An aggregated metric is also included.

For example:

```
# HELP jobs_row_level_ttl_total_rows Approximate number of rows on the TTL table.
# TYPE jobs_row_level_ttl_total_rows gauge
jobs_row_level_ttl_total_rows{node_id="1"} 0
jobs_row_level_ttl_total_rows{node_id="1",relation="default"} 0
jobs_row_level_ttl_total_rows{node_id="1",relation="defaultdb_public_events__events_pkey_"} 0
jobs_row_level_ttl_total_rows{node_id="1",relation="defaultdb_public_events_using_date__events_using_date_pkey_"} 0
```

Following is a list of the metrics that have multi-dimensional metrics:

<table className="crdb-fixed-table"><thead><tr><th style={{width: "30%"}}>CockroachDB Metric Name</th><th style={{width: "44%"}}>Description When Aggregated</th><th style={{width: "13%"}}>Type</th><th style={{width: "13%"}}>Unit</th></tr></thead><tbody> <tr><td><div id="jobs.row_level_ttl.delete_duration" class="anchored"><code>jobs.row\_level\_ttl.delete\_duration</code></div></td><td>Duration for delete requests during row level TTL.</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="jobs.row_level_ttl.num_active_spans" class="anchored"><code>jobs.row\_level\_ttl.num\_active\_spans</code></div></td><td>Number of active spans the TTL job is deleting from.</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="jobs.row_level_ttl.rows_deleted" class="anchored"><code>jobs.row\_level\_ttl.rows\_deleted</code></div></td><td>Number of rows deleted by the row level TTL job.</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="jobs.row_level_ttl.rows_selected" class="anchored"><code>jobs.row\_level\_ttl.rows\_selected</code></div></td><td>Number of rows selected for deletion by the row level TTL job.</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="jobs.row_level_ttl.select_duration" class="anchored"><code>jobs.row\_level\_ttl.select\_duration</code></div></td><td>Duration for select requests during row level TTL.</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="jobs.row_level_ttl.span_total_duration" class="anchored"><code>jobs.row\_level\_ttl.span\_total\_duration</code></div></td><td>Duration for processing a span during row level TTL.</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="jobs.row_level_ttl.total_expired_rows" class="anchored"><code>jobs.row\_level\_ttl.total\_expired\_rows</code></div></td><td>Approximate number of rows that have expired the TTL on the TTL table.</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="jobs.row_level_ttl.total_rows" class="anchored"><code>jobs.row\_level\_ttl.total\_rows</code></div></td><td>Approximate number of rows on the TTL table.</td><td>GAUGE</td><td>COUNT</td></tr> </tbody></table>

### Metrics of type histogram

When child metrics is enabled with changefeeds or row-level TTL jobs, be aware that metrics of type HISTOGRAM will increase cardinality quickly.

For example, when you create two changefeeds with the metrics labels `employees` and `office_dogs`, the histogram metric `changefeed_flush_hist_nanos` exports child metrics for each bucket for each metrics label. In addition, the `default` scope will also be exported which includes changefeeds started without a metrics label. Therefore, in this example, `changefeed_flush_hist_nanos` exports child metrics for each bucket for the scope values: `default`, `employees` and `office_dogs`:

```
# HELP changefeed_flush_hist_nanos Time spent flushing messages across all changefeeds
# TYPE changefeed_flush_hist_nanos histogram
changefeed_flush_hist_nanos_bucket{node_id="1",le="5e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",le="5.572592853587432e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",le="6.210758222370743e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",le="6.922005377068515e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",le="7.714703539349157e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",le="8.598180362184556e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",le="9.582831688033088e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.0680243876372875e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.1903330140009282e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.326648249442152e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.4785741108131227e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.6478983046833196e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.8366132632234223e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.046939589088547e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.2813521851760063e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.5426093767255764e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.833785368441068e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="3.1583064185550647e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="3.5199911554958534e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="3.9230955115614314e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="4.372362802333632e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="4.873079541115184e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="5.431137645156319e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="6.053103765649553e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="6.746296557296375e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="7.518872796674253e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="8.37992336275598e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="9.339580208980864e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.0409135585614676e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.1601174915283792e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.2929724885225649e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.4410418498852003e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.6060679028781363e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.789992503590971e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.9949798866972237e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.2234421319319225e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.478067546953807e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.7618523005723442e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="3.0781356785666904e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="3.4306393769506477e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="3.823511295046164e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="4.261374343677016e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="4.749380842807073e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="5.2932731487183495e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="5.899451224126824e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="6.575047946331352e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="7.328013039544164e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="8.167206619031862e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="9.102503447797786e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.0144909132590578e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.1306689626513626e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.2601515562088194e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.4044623113132697e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.5652993278314426e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.7445551695997415e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.9443391341601053e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.167002072794196e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.415164052912417e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.691745188300199e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="3.0000000000000085e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="+Inf"} 2
changefeed_flush_hist_nanos_sum{node_id="1"} 9.79696709e+08
changefeed_flush_hist_nanos_count{node_id="1"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="5e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="5.572592853587432e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="6.210758222370743e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="6.922005377068515e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="7.714703539349157e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="8.598180362184556e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="9.582831688033088e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.0680243876372875e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.1903330140009282e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.326648249442152e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.4785741108131227e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.6478983046833196e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.8366132632234223e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.046939589088547e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.2813521851760063e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.5426093767255764e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.833785368441068e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.1583064185550647e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.5199911554958534e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.9230955115614314e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="4.372362802333632e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="4.873079541115184e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="5.431137645156319e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="6.053103765649553e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="6.746296557296375e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="7.518872796674253e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="8.37992336275598e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="9.339580208980864e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.0409135585614676e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.1601174915283792e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.2929724885225649e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.4410418498852003e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.6060679028781363e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.789992503590971e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.9949798866972237e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.2234421319319225e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.478067546953807e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.7618523005723442e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.0781356785666904e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.4306393769506477e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.823511295046164e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="4.261374343677016e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="4.749380842807073e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="5.2932731487183495e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="5.899451224126824e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="6.575047946331352e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="7.328013039544164e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="8.167206619031862e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="9.102503447797786e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.0144909132590578e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.1306689626513626e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.2601515562088194e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.4044623113132697e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.5652993278314426e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.7445551695997415e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.9443391341601053e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.167002072794196e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.415164052912417e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.691745188300199e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.0000000000000085e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="+Inf"} 0
changefeed_flush_hist_nanos_sum{node_id="1",scope="default"} 0
changefeed_flush_hist_nanos_count{node_id="1",scope="default"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="5e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="5.572592853587432e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="6.210758222370743e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="6.922005377068515e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="7.714703539349157e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="8.598180362184556e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="9.582831688033088e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.0680243876372875e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.1903330140009282e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.326648249442152e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.4785741108131227e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.6478983046833196e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.8366132632234223e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.046939589088547e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.2813521851760063e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.5426093767255764e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.833785368441068e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.1583064185550647e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.5199911554958534e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.9230955115614314e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="4.372362802333632e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="4.873079541115184e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="5.431137645156319e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="6.053103765649553e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="6.746296557296375e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="7.518872796674253e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="8.37992336275598e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="9.339580208980864e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.0409135585614676e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.1601174915283792e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.2929724885225649e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.4410418498852003e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.6060679028781363e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.789992503590971e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.9949798866972237e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.2234421319319225e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.478067546953807e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.7618523005723442e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.0781356785666904e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.4306393769506477e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.823511295046164e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="4.261374343677016e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="4.749380842807073e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="5.2932731487183495e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="5.899451224126824e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="6.575047946331352e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="7.328013039544164e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="8.167206619031862e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="9.102503447797786e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.0144909132590578e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.1306689626513626e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.2601515562088194e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.4044623113132697e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.5652993278314426e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.7445551695997415e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.9443391341601053e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.167002072794196e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.415164052912417e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.691745188300199e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.0000000000000085e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="+Inf"} 0
changefeed_flush_hist_nanos_sum{node_id="1",scope="employees"} 0
changefeed_flush_hist_nanos_count{node_id="1",scope="employees"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="5e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="5.572592853587432e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="6.210758222370743e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="6.922005377068515e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="7.714703539349157e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="8.598180362184556e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="9.582831688033088e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.0680243876372875e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.1903330140009282e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.326648249442152e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.4785741108131227e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.6478983046833196e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.8366132632234223e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.046939589088547e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.2813521851760063e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.5426093767255764e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.833785368441068e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.1583064185550647e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.5199911554958534e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.9230955115614314e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="4.372362802333632e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="4.873079541115184e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="5.431137645156319e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="6.053103765649553e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="6.746296557296375e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="7.518872796674253e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="8.37992336275598e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="9.339580208980864e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.0409135585614676e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.1601174915283792e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.2929724885225649e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.4410418498852003e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.6060679028781363e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.789992503590971e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.9949798866972237e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.2234421319319225e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.478067546953807e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.7618523005723442e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.0781356785666904e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.4306393769506477e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.823511295046164e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="4.261374343677016e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="4.749380842807073e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="5.2932731487183495e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="5.899451224126824e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="6.575047946331352e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="7.328013039544164e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="8.167206619031862e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="9.102503447797786e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.0144909132590578e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.1306689626513626e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.2601515562088194e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.4044623113132697e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.5652993278314426e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.7445551695997415e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.9443391341601053e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.167002072794196e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.415164052912417e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.691745188300199e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.0000000000000085e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="+Inf"} 2
changefeed_flush_hist_nanos_sum{node_id="1",scope="office_dogs"} 9.79696709e+08
changefeed_flush_hist_nanos_count{node_id="1",scope="office_dogs"} 2
```

## Enable `database` and `application_name` labels

<Note>
  **This feature is in <InternalLink path="cockroachdb-feature-availability">preview</InternalLink>** and subject to change. To share feedback and/or issues, contact [Support](https://support.cockroachlabs.com).
</Note>

The following cluster settings enable the `database` and `application_name` labels for certain metrics, along with their internal counterparts if they exist:

* <InternalLink path="cluster-settings#setting-sql-metrics-database-name-enabled">`sql.metrics.database_name.enabled`</InternalLink>
* <InternalLink path="cluster-settings#setting-sql-metrics-application-name-enabled">`sql.metrics.application_name.enabled`</InternalLink>

By default, these cluster settings are disabled. To enable them, use the <InternalLink path="set-cluster-setting">`SET CLUSTER SETTING`</InternalLink> statement. Because these labels use aggregate metrics, you must enable the [`server.child_metrics.enabled`](#enable-child-metrics) cluster setting to use them.

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING server.child_metrics.enabled = true;
```

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING sql.metrics.database_name.enabled = true;
```

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING sql.metrics.application_name.enabled = true;
```

Toggling the `sql.metrics.database_name.enabled` and `sql.metrics.application_name.enabled` cluster settings clears existing metric values for current label combinations and reinitializes the affected metrics to reflect the new label configuration.

When toggling the `sql.metrics.database_name.enabled` and `sql.metrics.application_name.enabled` cluster settings, only the values for existing metric label combinations will be cleared. Aggregated metric values for the affected metrics will not be cleared.

<Note>
  Child metrics (metrics with the `database` and `application_name` labels) are independent from the parent (aggregated metric). The child metrics are initialized when the cluster settings are enabled.

  For this reason, child `COUNTER` metrics may not always add up to the parent `COUNTER` metric. For an example, refer to [Examples 1 through 6](#1-all-cluster-settings-disabled).

  For `GAUGE` metrics, values may be different and potentially unexpected depending on when a setting is enabled. For an example, refer to [7. `GAUGE` metric example](#7-gauge-metric-example).
</Note>

These labels affect only the metrics emitted via <InternalLink path="prometheus-endpoint">Prometheus export</InternalLink>. They are not visible in the <InternalLink path="monitoring-and-alerting#metrics-dashboards">DB Console Metrics dashboards</InternalLink>.

The system retains up to 5,000 recently used label combinations.

Following is a list of the metrics that have multi-dimensional metrics:

<table className="crdb-fixed-table"><thead><tr><th style={{width: "30%"}}>CockroachDB Metric Name</th><th style={{width: "44%"}}>Description When Aggregated</th><th style={{width: "13%"}}>Type</th><th style={{width: "13%"}}>Unit</th></tr></thead><tbody> <tr><td><div id="sql.crud_query.count" class="anchored"><code>sql.crud\_query.count</code></div></td><td>Number of SQL SELECT, INSERT, UPDATE, DELETE statements successfully executed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.crud_query.count.internal" class="anchored"><code>sql.crud\_query.count.internal</code></div></td><td>Number of SQL SELECT, INSERT, UPDATE, DELETE statements successfully executed (internal queries)</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.delete.count" class="anchored"><code>sql.delete.count</code></div></td><td>Number of SQL DELETE statements successfully executed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.delete.count.internal" class="anchored"><code>sql.delete.count.internal</code></div></td><td>Number of SQL DELETE statements successfully executed (internal queries)</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.distsql.contended_queries.count" class="anchored"><code>sql.distsql.contended\_queries.count</code></div></td><td>Number of SQL queries that experienced contention</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.distsql.cumulative_contention_nanos" class="anchored"><code>sql.distsql.cumulative\_contention\_nanos</code></div></td><td>Cumulative contention across all queries (in nanoseconds)</td><td>COUNTER</td><td>NANOSECONDS</td></tr> <tr><td><div id="sql.failure.count" class="anchored"><code>sql.failure.count</code></div></td><td>Number of statements resulting in a planning or runtime error</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.failure.count.internal" class="anchored"><code>sql.failure.count.internal</code></div></td><td>Number of statements resulting in a planning or runtime error (internal queries)</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.full.scan.count" class="anchored"><code>sql.full.scan.count</code></div></td><td>Number of full table or index scans</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.full.scan.count.internal" class="anchored"><code>sql.full.scan.count.internal</code></div></td><td>Number of full table or index scans (internal queries)</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.insert.count" class="anchored"><code>sql.insert.count</code></div></td><td>Number of SQL INSERT statements successfully executed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.insert.count.internal" class="anchored"><code>sql.insert.count.internal</code></div></td><td>Number of SQL INSERT statements successfully executed (internal queries)</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.select.count" class="anchored"><code>sql.select.count</code></div></td><td>Number of SQL SELECT statements successfully executed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.select.count.internal" class="anchored"><code>sql.select.count.internal</code></div></td><td>Number of SQL SELECT statements successfully executed (internal queries)</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.service.latency" class="anchored"><code>sql.service.latency</code></div></td><td>Latency of SQL request execution</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="sql.service.latency.internal" class="anchored"><code>sql.service.latency.internal</code></div></td><td>Latency of SQL request execution (internal queries)</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="sql.statements.active" class="anchored"><code>sql.statements.active</code></div></td><td>Number of currently active user SQL statements</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="sql.statements.active.internal" class="anchored"><code>sql.statements.active.internal</code></div></td><td>Number of currently active user SQL statements (internal queries)</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="sql.txn.begin.count" class="anchored"><code>sql.txn.begin.count</code></div></td><td>Number of SQL transaction BEGIN statements successfully executed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.txn.begin.count.internal" class="anchored"><code>sql.txn.begin.count.internal</code></div></td><td>Number of SQL transaction BEGIN statements successfully executed (internal queries)</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.txn.commit.count" class="anchored"><code>sql.txn.commit.count</code></div></td><td>Number of SQL transaction COMMIT statements successfully executed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.txn.commit.count.internal" class="anchored"><code>sql.txn.commit.count.internal</code></div></td><td>Number of SQL transaction COMMIT statements successfully executed (internal queries)</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.txn.latency" class="anchored"><code>sql.txn.latency</code></div></td><td>Latency of SQL transactions</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="sql.txn.latency.internal" class="anchored"><code>sql.txn.latency.internal</code></div></td><td>Latency of SQL transactions (internal queries)</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> <tr><td><div id="sql.txn.rollback.count" class="anchored"><code>sql.txn.rollback.count</code></div></td><td>Number of SQL transaction ROLLBACK statements successfully executed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.txn.rollback.count.internal" class="anchored"><code>sql.txn.rollback.count.internal</code></div></td><td>Number of SQL transaction ROLLBACK statements successfully executed (internal queries)</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.txns.open" class="anchored"><code>sql.txns.open</code></div></td><td>Number of currently open user SQL transactions</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="sql.txns.open.internal" class="anchored"><code>sql.txns.open.internal</code></div></td><td>Number of currently open user SQL transactions (internal queries)</td><td>GAUGE</td><td>COUNT</td></tr> <tr><td><div id="sql.update.count" class="anchored"><code>sql.update.count</code></div></td><td>Number of SQL UPDATE statements successfully executed</td><td>COUNTER</td><td>COUNT</td></tr> <tr><td><div id="sql.update.count.internal" class="anchored"><code>sql.update.count.internal</code></div></td><td>Number of SQL UPDATE statements successfully executed (internal queries)</td><td>COUNTER</td><td>COUNT</td></tr> </tbody></table>

### Examples

This section demonstrates the impact of enabling and disabling the relevant cluster settings.

[Examples 1 through 6](#1-all-cluster-settings-disabled) show the effect on the `COUNTER` metric `sql.select.count`. During these examples, the <InternalLink path="cockroach-workload#run-the-movr-workload">`movr` workload</InternalLink> was running on node 1. The aggregated metric consistently increases as the examples progress.

[Example 7](#7-gauge-metric-example) shows a possible effect on the `GAUGE` metric `sql.txn.open`.

#### 1. All cluster settings disabled

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING server.child_metrics.enabled = false;
SET CLUSTER SETTING sql.metrics.database_name.enabled = false;
SET CLUSTER SETTING sql.metrics.application_name.enabled = false;
```

The Prometheus export only gives the aggregated metric for the node.

```
# HELP sql_select_count Number of SQL SELECT statements successfully executed
# TYPE sql_select_count counter
sql_select_count{node_id="1"} 2030
```

#### 2. Only child metrics enabled

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING server.child_metrics.enabled = true;
SET CLUSTER SETTING sql.metrics.database_name.enabled = false;
SET CLUSTER SETTING sql.metrics.application_name.enabled = false;
```

The Prometheus export still only gives the aggregated metric for the node.

```
sql_select_count{node_id="1"} 6568
```

#### 3. Child metrics and `database_name` label enabled

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING server.child_metrics.enabled = true;
SET CLUSTER SETTING sql.metrics.database_name.enabled = true;
SET CLUSTER SETTING sql.metrics.application_name.enabled = false;
```

The aggregated metric and a child metric with only the `database` label are emitted.

```
sql_select_count{node_id="1"} 10259
sql_select_count{node_id="1",database="movr"} 816
```

#### 4. Child metrics and `application_name` label enabled

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING server.child_metrics.enabled = true;
SET CLUSTER SETTING sql.metrics.database_name.enabled = false;
SET CLUSTER SETTING sql.metrics.application_name.enabled = true;
```

The aggregated metric and a child metric with only the `application_name` label are emitted. Note that even though the aggregated metric has increased, the child metric with `application_name` label has a value less than the child metric with `database` label in the [preceding example](#3-child-metrics-and-database_name-label-enabled). This is because the labeled metrics have been reset, while the aggregated metric was not reset.

```
sql_select_count{node_id="1"} 14077
sql_select_count{node_id="1",application_name="movr"} 718
```

#### 5. Child metrics and both `database` and `application_name` label enabled

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING server.child_metrics.enabled = true;
SET CLUSTER SETTING sql.metrics.database_name.enabled = true;
SET CLUSTER SETTING sql.metrics.application_name.enabled = true;
```

The aggregated metric and a child metric with both `database` and `application_name` labels are emitted.

```
sql_select_count{node_id="1"} 21085
sql_select_count{node_id="1",database="movr",application_name="movr"} 3962
```

#### 6. Aggregate metric disabled

The <InternalLink path="cluster-settings#setting-server-child-metrics-include-aggregate-enabled">cluster setting `server.child_metrics.include_aggregate.enabled`</InternalLink> (default: `true`) reports an aggregate time series for applicable multi-dimensional metrics. When set to `false`, it stops reporting the aggregate time series, preventing double counting when querying those metrics.

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING server.child_metrics.enabled = true;
SET CLUSTER SETTING sql.metrics.database_name.enabled = true;
SET CLUSTER SETTING sql.metrics.application_name.enabled = true;
SET CLUSTER SETTING server.child_metrics.include_aggregate.enabled = false;
```

No aggregated metric emitted.
Only the child metric with both the `database` and `application_name` labels is emitted.

```
sql_select_count{node_id="1",database="movr",application_name="movr"} 8703
```

#### 7. `GAUGE` metric example

Changes to cluster settings may take time to reinitialize affected metrics. As a result, some `GAUGE` metrics might briefly show unexpected values.

`GAUGE` values for both aggregated and child metrics increase and decrease as transactions are opened and closed. This example uses the `GAUGE` metric `sql.txn.open`.

Consider the following scenario:

| Time | Action                                                                                                                   | `sql.txn.open` aggregated metric | `sql.txn.open` child metric |
| :--: | ------------------------------------------------------------------------------------------------------------------------ | :------------------------------: | :-------------------------: |
|   1  | Open a transaction. The value of the `sql.txn.open` aggregated metric is incremented.                                    |                 1                |              -              |
|   2  | Enable `sql.metrics.database_name.enabled` and `sql.metrics.application_name.enabled`.<br />Re-initialize child metrics. |                 1                |              0              |
|   3  | Close a transaction.<br />The values of both the aggregated and child `sql.txn.open` metrics are decremented.            |                 0                |              -1             |

To avoid negative values in child metrics, use the [Prometheus `clamp_min` function](https://prometheus.io/docs/prometheus/latest/querying/functions/#clamp_min) to set the metric to zero.

## Enable detailed latency metrics

The <InternalLink path="cluster-settings#setting-sql-stats-detailed-latency-metrics-enabled">cluster setting `sql.stats.detailed_latency_metrics.enabled`</InternalLink> labels the latency metric `sql.exec.latency.detail` with the <InternalLink path="ui-statements-page#sql-statement-fingerprints">statement fingerprint</InternalLink>. To estimate the cardinality of the set of all statement fingerprints, use the `sql.query.unique.count` metric. For most workloads, this metric ranges from dozens to hundreds.

`sql.stats.detailed_latency_metrics.enabled` is disabled by default. To enable it, use the <InternalLink path="set-cluster-setting">`SET CLUSTER SETTING`</InternalLink> statement.

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING sql.stats.detailed_latency_metrics.enabled = true;
```

<Danger>
  Be aware that the `sql.exec.latency.detail` metric is of type HISTOGRAM and will quickly increase the cardinality of exported metrics. For workloads with over a couple thousand statement fingerprints, exercise caution when enabling `sql.stats.detailed_latency_metrics.enabled`. For workloads with tens of thousands of distinct query fingerprints, leave this cluster setting set to `false`.
</Danger>

Following is a list of the metrics that have multi-dimensional metrics:

<table className="crdb-fixed-table"><thead><tr><th style={{width: "30%"}}>CockroachDB Metric Name</th><th style={{width: "44%"}}>Description</th><th style={{width: "13%"}}>Type</th><th style={{width: "13%"}}>Unit</th></tr></thead><tbody> <tr><td><div id="sql.exec.latency.detail" class="anchored"><code>sql.exec.latency.detail</code></div></td><td>Latency of SQL statement execution, by statement fingerprint</td><td>HISTOGRAM</td><td>NANOSECONDS</td></tr> </tbody></table>

For example:

```
# HELP sql_query_unique_count Cardinality estimate of the set of statement fingerprints
# TYPE sql_query_unique_count counter
sql_query_unique_count{node_id="1"} 48
...
# HELP sql_exec_latency_detail Latency of SQL statement execution, by statement fingerprint
# TYPE sql_exec_latency_detail histogram
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="10000"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="12638.482029342978"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="15973.122800602541"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="20187.602546790382"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="25514.065200312878"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="32245.90545296394"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="40753.92965871775"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="51506.78076168121"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="65096.75230458167"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="82272.41341700466"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="103979.84184814895"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="131414.73626117557"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="166088.27826277143"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="209910.3720108553"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="265294.8464431894"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="335292.41492495546"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="423758.7160604059"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="535566.6917706894"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="676875.0009458527"} 0
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="855467.2535565672"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.0811807510766068e+06"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.3664483492953242e+06"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.7269832906594332e+06"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="2.182644728397485e+06"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="2.7585316176291797e+06"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="3.4863652276780806e+06"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="4.406236427773566e+06"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="5.568813990945262e+06"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="7.038135554931545e+06"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="8.89513497310822e+06"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.1242100350620849e+07"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.4208308325339198e+07"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.795714494371637e+07"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="2.269510536694665e+07"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="2.868316813342006e+07"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="3.625117049988527e+07"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="4.581597669054482e+07"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="5.790443980602476e+07"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="7.318242219076161e+07"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="9.249147277217315e+07"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.1689518164985757e+08"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.4773776525985083e+08"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.8671810912919158e+08"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="2.359833466782189e+08"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="2.9824712862168837e+08"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="3.769390975388353e+08"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="4.7639380104013294e+08"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="6.020894493336115e+08"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="7.609496685459859e+08"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="9.61724871115294e+08"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.2154742500762835e+09"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.5361749466718242e+09"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.941491945743876e+09"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="2.453751106639811e+09"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="3.10116892657477e+09"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="3.919406774847209e+09"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="4.953535208959157e+09"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="6.260516572014802e+09"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="7.912342618981298e+09"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="9.99999999999997e+09"} 1
sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="+Inf"} 1
sql_exec_latency_detail_sum{fingerprint="SHOW database",node_id="1"} 723750
sql_exec_latency_detail_count{fingerprint="SHOW database",node_id="1"} 1
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="10000"} 0
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="12638.482029342978"} 0
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="15973.122800602541"} 0
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="20187.602546790382"} 0
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="25514.065200312878"} 0
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="32245.90545296394"} 0
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="40753.92965871775"} 55
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="51506.78076168121"} 319
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="65096.75230458167"} 801
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="82272.41341700466"} 1503
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="103979.84184814895"} 2307
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="131414.73626117557"} 2975
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="166088.27826277143"} 3439
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="209910.3720108553"} 3779
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="265294.8464431894"} 3912
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="335292.41492495546"} 4039
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="423758.7160604059"} 4156
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="535566.6917706894"} 4236
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="676875.0009458527"} 4293
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="855467.2535565672"} 4308
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.0811807510766068e+06"} 4323
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.3664483492953242e+06"} 4330
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.7269832906594332e+06"} 4343
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="2.182644728397485e+06"} 4347
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="2.7585316176291797e+06"} 4348
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="3.4863652276780806e+06"} 4349
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="4.406236427773566e+06"} 4349
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="5.568813990945262e+06"} 4349
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="7.038135554931545e+06"} 4349
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="8.89513497310822e+06"} 4349
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.1242100350620849e+07"} 4350
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.4208308325339198e+07"} 4350
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.795714494371637e+07"} 4350
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="2.269510536694665e+07"} 4351
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="2.868316813342006e+07"} 4354
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="3.625117049988527e+07"} 4359
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="4.581597669054482e+07"} 4363
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="5.790443980602476e+07"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="7.318242219076161e+07"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="9.249147277217315e+07"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.1689518164985757e+08"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.4773776525985083e+08"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.8671810912919158e+08"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="2.359833466782189e+08"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="2.9824712862168837e+08"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="3.769390975388353e+08"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="4.7639380104013294e+08"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="6.020894493336115e+08"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="7.609496685459859e+08"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="9.61724871115294e+08"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.2154742500762835e+09"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.5361749466718242e+09"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.941491945743876e+09"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="2.453751106639811e+09"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="3.10116892657477e+09"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="3.919406774847209e+09"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="4.953535208959157e+09"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="6.260516572014802e+09"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="7.912342618981298e+09"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="9.99999999999997e+09"} 4367
sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="+Inf"} 4367
sql_exec_latency_detail_sum{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1"} 1.27599889e+09
sql_exec_latency_detail_count{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1"} 4367
```