> ## Documentation Index > Fetch the complete documentation index at: https://docs.cockroachlabs.com/llms.txt > Use this file to discover all available pages before exploring further. # Multi-dimensional Metrics Multi-dimensional metrics are additional [Prometheus](/docs/stable/prometheus-endpoint.md) time series with extra labels. This page will help you understand the potential size of the Prometheus scrape payload for your workload when multi-dimensional metrics are enabled. The number of multi-dimensional metrics can significantly increase based on their associated labels, which increases cardinality. The export of multi-dimensional metrics can be enabled by two [cluster settings](/docs/stable/cluster-settings.md): * [`server.child_metrics.enabled`](#enable-child-metrics) * [`sql.metrics.database_name.enabled`](#enable-database-and-application_name-labels) * [`sql.metrics.application_name.enabled`](#enable-database-and-application_name-labels) * [`sql.stats.detailed_latency_metrics.enabled`](#enable-detailed-latency-metrics) ## Enable child metrics Child metrics are specific, detailed metrics that are usually related to a higher-level (parent or aggregate) metric. They often provide more granular or specific information about a particular aspect of the parent metric. The parent metrics and their potential child metrics are determined by the specific feature the cluster is using. The [cluster setting `server.child_metrics.enabled`](/docs/stable/cluster-settings#setting-server-child-metrics-enabled) is disabled by default. To enable it, use the [`SET CLUSTER SETTING`](/docs/stable/set-cluster-setting) statement. ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} SET CLUSTER SETTING server.child_metrics.enabled = true; ``` The [cluster setting `server.child_metrics.include_aggregate.enabled`](/docs/stable/cluster-settings#setting-server-child-metrics-include-aggregate-enabled) (default: `true`) reports an aggregate time series for applicable multi-dimensional metrics. When set to `false`, it stops reporting the aggregate time series, preventing double counting when querying those metrics. ### All clusters An RPC (Remote Procedure Call) connection is a communication method used in distributed systems, like CockroachDB, to allow one program to request a service from a program located in another computer on a network without having to understand the network's details. In the context of CockroachDB, RPC connections are used for inter-node communication. For instance, if Node 1 sends a request to Node 2, and Node 2 dials back (sends request back to Node 1), it ensures that communication is healthy in both directions. This is referred to as a "bidirectionally connected" and "heartbeating" RPC connection. When child metrics is enabled, for all clusters the `rpc.connection.*` metrics are exported per-peer with labels for `remote_node_id`, `remote_addr`, and `class`. The `class` label may have the following values: `system`, `default`, and `raft`. The cardinality increases with the number of nodes. An aggregated metric is also included. For example: ``` # HELP rpc_connection_healthy Gauge of current connections in a healthy state (i.e. bidirectionally connected and heartbeating) # TYPE rpc_connection_healthy gauge rpc_connection_healthy{node_id="1"} 10 rpc_connection_healthy{node_id="1",remote_node_id="0",remote_addr="127.0.0.1:29004",class="system"} 1 rpc_connection_healthy{node_id="1",remote_node_id="0",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29000",class="system"} 0 rpc_connection_healthy{node_id="1",remote_node_id="0",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29002",class="system"} 1 rpc_connection_healthy{node_id="1",remote_node_id="1",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29004",class="default"} 1 rpc_connection_healthy{node_id="1",remote_node_id="1",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29004",class="system"} 1 rpc_connection_healthy{node_id="1",remote_node_id="2",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29002",class="default"} 1 rpc_connection_healthy{node_id="1",remote_node_id="2",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29002",class="raft"} 1 rpc_connection_healthy{node_id="1",remote_node_id="2",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29002",class="system"} 1 rpc_connection_healthy{node_id="1",remote_node_id="3",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29000",class="default"} 1 rpc_connection_healthy{node_id="1",remote_node_id="3",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29000",class="raft"} 1 rpc_connection_healthy{node_id="1",remote_node_id="3",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29000",class="system"} 1 ``` Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
`rpc.connection.avg_round_trip_latency`Sum of exponentially weighted moving average of round-trip latencies, as measured through a gRPC RPC. Since this metric is based on gRPC RPCs, it is affected by application-level processing delays and CPU overload effects. See rpc.connection.tcp\_rtt for a metric that is obtained from the kernel's TCP stack. Dividing this Gauge by rpc.connection.healthy gives an approximation of average latency, but the top-level round-trip-latency histogram is more useful. Instead, users should consult the label families of this metric if they are available (which requires prometheus and the cluster setting 'server.child\_metrics.enabled'); these provide per-peer moving averages. This metric does not track failed connection. A failed connection's contribution is reset to zero.GAUGENANOSECONDS
`rpc.connection.failures`Counter of failed connections. This includes both the event in which a healthy connection terminates as well as unsuccessful reconnection attempts. Connections that are terminated as part of local node shutdown are excluded. Decommissioned peers are excluded.COUNTERCOUNT
`rpc.connection.healthy`Gauge of current connections in a healthy state (i.e. bidirectionally connected and heartbeating)GAUGECOUNT
`rpc.connection.healthy_nanos`Gauge of nanoseconds of healthy connection time On the prometheus endpoint scraped with the cluster setting 'server.child\_metrics.enabled' set, the constituent parts of this metric are available on a per-peer basis and one can read off for how long a given peer has been connectedGAUGENANOSECONDS
`rpc.connection.heartbeats`Counter of successful heartbeats.COUNTERCOUNT
`rpc.connection.inactive`Gauge of current connections in an inactive state and pending deletion; these are not healthy but are not tracked as unhealthy either because there is reason to believe that the connection is no longer relevant,for example if the node has since been seen under a new addressGAUGECOUNT
`rpc.connection.unhealthy`Gauge of current connections in an unhealthy state (not bidirectionally connected or heartbeating)GAUGECOUNT
`rpc.connection.unhealthy_nanos`Gauge of nanoseconds of unhealthy connection time. On the prometheus endpoint scraped with the cluster setting 'server.child\_metrics.enabled' set, the constituent parts of this metric are available on a per-peer basis and one can read off for how long a given peer has been unreachableGAUGENANOSECONDS
### Secure clusters When child metrics is enabled, for [secure clusters](/docs/stable/secure-a-cluster.md) the `security.certificate.expiration.client` is exported per SQL user with a label for `sql_user`. The `sql_user` label may have the values of the cluster's users who are logged into a node using client security certificates. The cardinality increases with the number of SQL users. An aggregated metric is also included, however since this is a sum of the child metric values which represent timestamps, it is not usable. For example: ``` # HELP security_certificate_expiration_client Minimum expiration for client certificates, labeled by SQL user. 0 means no certificate or error. # TYPE security_certificate_expiration_client gauge security_certificate_expiration_client{node_id="1"} 3.756110141e+09 security_certificate_expiration_client{node_id="1",sql_user="florence"} 1.878055127e+09 security_certificate_expiration_client{node_id="1",sql_user="root"} 1.878055014e+09 ``` Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
`security.certificate.expiration.client`Minimum expiration for client certificates, labeled by SQL user. 0 means no certificate or error.GAUGETIMESTAMP\_SEC
### Virtual clusters When child metrics is enabled, for [virtual clusters](/docs/stable/cluster-virtualization-overview.md) the `kv.tenant_rate_limit.*` metrics and other kv-related metrics are exported per virtual cluster with a label for `tenant_id`. The `tenant_id` label may have the values: `system` or the id of the virtual cluster. The cardinality increases with the number of virtual clusters. An aggregated metric is also included. > [!NOTE] > > With virtual clusters, while the `tenant_id` label on kv metrics is only exported when child metrics is enabled, [the `tenant` label on SQL metrics](/docs/stable/work-with-virtual-clusters#work-with-metrics-with-cluster-virtualization-enabled) is exported whether child metrics is enabled or disabled. For example: ``` # HELP kv_tenant_rate_limit_read_bytes_admitted Number of read bytes admitted by the rate limiter # TYPE kv_tenant_rate_limit_read_bytes_admitted counter kv_tenant_rate_limit_read_bytes_admitted{store="1",node_id="1"} 41 kv_tenant_rate_limit_read_bytes_admitted{store="1",node_id="1",tenant_id="3"} 41 kv_tenant_rate_limit_read_bytes_admitted{store="1",node_id="1",tenant_id="4"} 0 kv_tenant_rate_limit_read_bytes_admitted{store="1",node_id="1",tenant_id="system"} 0 # HELP sysbytes Number of bytes in system KV pairs # TYPE sysbytes gauge sysbytes{store="1",node_id="1"} 41979 sysbytes{store="1",node_id="1",tenant_id="3"} 278 sysbytes{store="1",node_id="1",tenant_id="4"} 225 sysbytes{store="1",node_id="1",tenant_id="system"} 41476 ``` Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
`abortspanbytes`Number of bytes in the abort spanGAUGEBYTES
`gcbytesage`Cumulative age of non-live dataGAUGESECONDS
`intentage`Cumulative age of locksGAUGESECONDS
`intentbytes`Number of bytes in intent KV pairsGAUGEBYTES
`intentcount`Count of intent keysGAUGECOUNT
`keybytes`Number of bytes taken up by keysGAUGEBYTES
`keycount`Count of all keysGAUGECOUNT
`kv.tenant_rate_limit.current_blocked`Number of requests currently blocked by the rate limiterGAUGECOUNT
`kv.tenant_rate_limit.num_tenants`Number of tenants currently being trackedGAUGECOUNT
`kv.tenant_rate_limit.read_batches_admitted`Number of read batches admitted by the rate limiterCOUNTERCOUNT
`kv.tenant_rate_limit.read_bytes_admitted`Number of read bytes admitted by the rate limiterCOUNTERBYTES
`kv.tenant_rate_limit.read_requests_admitted`Number of read requests admitted by the rate limiterCOUNTERCOUNT
`kv.tenant_rate_limit.write_batches_admitted`Number of write batches admitted by the rate limiterCOUNTERCOUNT
`kv.tenant_rate_limit.write_bytes_admitted`Number of write bytes admitted by the rate limiterCOUNTERBYTES
`kv.tenant_rate_limit.write_requests_admitted`Number of write requests admitted by the rate limiterCOUNTERCOUNT
`livebytes`Number of bytes of live data (keys plus values)GAUGEBYTES
`livecount`Count of live keysGAUGECOUNT
`lockbytes`Number of bytes taken up by replicated lock key-values (shared and exclusive strength, not intent strength)GAUGEBYTES
`lockcount`Count of replicated locks (shared, exclusive, and intent strength)GAUGECOUNT
`rangekeybytes`Number of bytes taken up by range keys (e.g. MVCC range tombstones)GAUGEBYTES
`rangekeycount`Count of all range keys (e.g. MVCC range tombstones)GAUGECOUNT
`rangevalbytes`Number of bytes taken up by range key values (e.g. MVCC range tombstones)GAUGEBYTES
`rangevalcount`Count of all range key values (e.g. MVCC range tombstones)GAUGECOUNT
`sysbytes`Number of bytes in system KV pairsGAUGEBYTES
`syscount`Count of system KV pairsGAUGECOUNT
`totalbytes`Total number of bytes taken up by keys and values including non-live dataGAUGEBYTES
`valbytes`Number of bytes taken up by valuesGAUGEBYTES
`valcount`Count of all valuesGAUGECOUNT
### Clusters with changefeeds When child metrics is enabled and [changefeeds with metrics labels](/docs/stable/monitor-and-debug-changefeeds.md#using-changefeed-metrics-labels) are created on the cluster, the `changefeed.*` metrics are exported per changefeed metric label with a label for `scope`. The `scope` label may have the values set using the `metrics_label` option. The cardinality increases with the number of changefeed metric labels. An aggregated metric is also included. For example, when you create two changefeeds with the metrics labels `employees` and `office_dogs`, the counter metric `changefeed_error_retries` exports child metrics with a `scope` for `employees` and `office_dogs`. In addition, the `default` scope will also be exported which includes changefeeds started without a metrics label. ``` HELP changefeed_error_retries Total retryable errors encountered by all changefeeds # TYPE changefeed_error_retries counter changefeed_error_retries{node_id="1"} 0 changefeed_error_retries{node_id="1",scope="default"} 0 changefeed_error_retries{node_id="1",scope="employees"} 0 changefeed_error_retries{node_id="1",scope="office_dogs"} 0 ``` Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
`changefeed.admit_latency`Event admission latency: a difference between event MVCC timestamp and the time it was admitted into changefeed pipeline; Note: this metric includes the time spent waiting until event can be processed due to backpressure or time spent resolving schema descriptors. Also note, this metric excludes latency during backfillHISTOGRAMNANOSECONDS
`changefeed.aggregator_progress`The earliest timestamp up to which any aggregator is guaranteed to have emitted all values forGAUGETIMESTAMP\_NS
`changefeed.backfill_count`Number of changefeeds currently executing backfillGAUGECOUNT
`changefeed.backfill_pending_ranges`Number of ranges in an ongoing backfill that are yet to be fully emittedGAUGECOUNT
`changefeed.batch_reduction_count`Number of times a Kafka changefeed split an over-large message batch and re-sent it after the broker rejected it (see changefeed.batch\_reduction\_retry.enabled). Emitted regardless of sink batching implementation.GAUGECOUNT
`changefeed.checkpoint_progress`The earliest timestamp of any changefeed's persisted checkpoint (values prior to this timestamp will never need to be re-emitted)GAUGETIMESTAMP\_NS
`changefeed.cloudstorage_buffered_bytes`The number of bytes buffered in cloudstorage sink files which have not been emitted yetGAUGECOUNT
`changefeed.commit_latency`Event commit latency: a difference between event MVCC timestamp and the time it was acknowledged by the downstream sink. If the sink batches events, then the difference between the oldest event in the batch and acknowledgement is recorded. Excludes latency during backfill.HISTOGRAMNANOSECONDS
`changefeed.emitted_batch_sizes`Size of batches emitted emitted by all feedsHISTOGRAMCOUNT
`changefeed.emitted_bytes`Bytes emitted by all feedsCOUNTERBYTES
`changefeed.emitted_messages`Messages emitted by all feedsCOUNTERCOUNT
`changefeed.error_retries`Total retryable errors encountered by all changefeedsCOUNTERCOUNT
`changefeed.filtered_messages`Messages filtered out by all feeds. This count does not include the number of messages that may be filtered due to the range constraints.COUNTERCOUNT
`changefeed.flush_hist_nanos`Time spent flushing messages across all changefeedsHISTOGRAMNANOSECONDS
`changefeed.flushed_bytes`Bytes emitted by all feeds; maybe different from changefeed.emitted\_bytes when compression is enabledCOUNTERBYTES
`changefeed.flushes`Total flushes across all feeds.COUNTERCOUNT
`changefeed.internal_retry_message_count`Number of messages re-sent by the batching sink after a retryable send error. Not emitted by the default Kafka, Pub/Sub, and webhook sinks.GAUGECOUNT
`changefeed.kafka_throttling_hist_nanos`Time spent in throttling due to exceeding kafka quotaHISTOGRAMNANOSECONDS
`changefeed.lagging_ranges`The number of ranges considered to be lagging behindGAUGECOUNT
`changefeed.max_behind_nanos`The most any changefeed's persisted checkpoint is behind the presentGAUGENANOSECONDS
`changefeed.message_size_hist`Message size histogramHISTOGRAMBYTES
`changefeed.parallel_io_in_flight_keys`The number of keys currently in-flight which may contend with batches pending to be emitted. Not emitted by the default Kafka, Pub/Sub, and webhook sinks.GAUGECOUNT
`changefeed.parallel_io_pending_rows`Number of rows which are blocked from being sent due to conflicting in-flight keys. Not emitted by the default Kafka, Pub/Sub, and webhook sinks.GAUGECOUNT
`changefeed.parallel_io_queue_nanos`Time that outgoing requests to the sink spend waiting in a queue due to in-flight requests with conflicting keys. Not emitted by the default Kafka, Pub/Sub, and webhook sinks.HISTOGRAMNANOSECONDS
`changefeed.parallel_io_result_queue_nanos`Time that incoming results from the sink spend waiting in parallel io emitter before they are acknowledged by the changefeed. Not emitted by the default Kafka, Pub/Sub, and webhook sinks.HISTOGRAMNANOSECONDS
`changefeed.parallel_io_workers`The number of workers in the ParallelIO. Not emitted by the default Kafka, Pub/Sub, and webhook sinks.GAUGECOUNT
`changefeed.progress_skew.span`The time difference between the fastest and slowest span's resolved timestampGAUGENANOSECONDS
`changefeed.progress_skew.table`The time difference between the fastest and slowest table's resolved timestampGAUGENANOSECONDS
`changefeed.running`Number of currently running changefeeds, including sinklessGAUGECOUNT
`changefeed.schema_registry.registrations`Number of registration attempts with the schema registry.COUNTERCOUNT
`changefeed.schema_registry.retry_count`Number of retries encountered when sending requests to the schema registry.COUNTERCOUNT
`changefeed.sink_backpressure_nanos`Time spent waiting for quota when emitting to the sink (back-pressure). Not emitted by the default Kafka, Pub/Sub, and webhook sinks.HISTOGRAMNANOSECONDS
`changefeed.sink_batch_hist_nanos`Time spent batched in the sink buffer before being flushed and acknowledged. Not emitted by the default Kafka, Pub/Sub, and webhook sinks.HISTOGRAMNANOSECONDS
`changefeed.sink_io_inflight`The number of messages currently inflight as IO requests being sent to the sinkGAUGECOUNT
`changefeed.size_based_flushes`Total size based flushes across all feeds. Not emitted by the default Kafka, Pub/Sub, and webhook sinks.COUNTERCOUNT
### Clusters with logical data replication jobs When child metrics is enabled and [logical data replication (LDR) jobs with metrics labels](/docs/stable/logical-data-replication-monitoring.md#metrics-labels) are created on the cluster, the `logical_replication_*_by_label` metrics are exported per LDR metric label. The `label` may have the values set using the `label` option. The cardinality increases with the number of LDR metric labels. For example, when you create two LDR jobs with the metrics labels `ldr_job1` and `ldr_job2`, the metrics `logical_replication_*_by_label` export child metrics with a `label` for `ldr_job1` and `ldr_job2`. ``` # HELP logical_replication_replicated_time_by_label Replicated time of the logical replication stream by label # TYPE logical_replication_replicated_time_by_label gauge logical_replication_replicated_time_by_label{label="ldr_job2",node_id="2"} 1.73411035e+09 logical_replication_replicated_time_by_label{label="ldr_job1",node_id="2"} 1.73411035e+09 # HELP logical_replication_catchup_ranges_by_label Source side ranges undergoing catch up scans # TYPE logical_replication_catchup_ranges_by_label gauge logical_replication_catchup_ranges_by_label{label="ldr_job1",node_id="2"} 0 logical_replication_catchup_ranges_by_label{label="ldr_job2",node_id="2"} 0 # HELP logical_replication_scanning_ranges_by_label Source side ranges undergoing an initial scan # TYPE logical_replication_scanning_ranges_by_label gauge logical_replication_scanning_ranges_by_label{label="ldr_job1",node_id="2"} 0 logical_replication_scanning_ranges_by_label{label="ldr_job2",node_id="2"} 0 ``` Note that the `logical_replication_*` metrics without the `_by_label` suffix may be `inaccurate with multiple LDR jobs`. ``` # HELP logical_replication_catchup_ranges Source side ranges undergoing catch up scans (inaccurate with multiple LDR jobs) # TYPE logical_replication_catchup_ranges gauge logical_replication_catchup_ranges{node_id="2"} 0 # HELP logical_replication_scanning_ranges Source side ranges undergoing an initial scan (inaccurate with multiple LDR jobs) # TYPE logical_replication_scanning_ranges gauge logical_replication_scanning_ranges{node_id="2"} 0 ``` Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescriptionTypeUnit
`logical_replication.catchup_ranges_by_label`Source side ranges undergoing catch up scansGAUGECOUNT
`logical_replication.events_dlqed_by_label`Row update events sent to DLQ by labelCOUNTERCOUNT
`logical_replication.events_ingested_by_label`Events ingested by all replication jobs by labelCOUNTERCOUNT
`logical_replication.replicated_time_by_label`Replicated time of the logical replication stream by labelGAUGESECONDS
`logical_replication.scanning_ranges_by_label`Source side ranges undergoing an initial scanGAUGECOUNT
### Clusters with row-level TTL jobs When child metrics is enabled and [row-level TTL jobs](/docs/stable/row-level-ttl.md#ttl-metrics) are created on the cluster with the [`ttl_label_metrics` storage parameter enabled](/docs/stable/row-level-ttl#ttl-metrics), the `jobs.row_level_ttl.*` metrics are exported per TTL job with `ttl_label_metrics` enabled with a label for `relation`. The value of the `relation` label may have the format: `{database}_{schema}_{table}_{primary key}`. The cardinality increases with the number of TTL jobs with `ttl_label_metrics` enabled. An aggregated metric is also included. For example: ``` # HELP jobs_row_level_ttl_total_rows Approximate number of rows on the TTL table. # TYPE jobs_row_level_ttl_total_rows gauge jobs_row_level_ttl_total_rows{node_id="1"} 0 jobs_row_level_ttl_total_rows{node_id="1",relation="default"} 0 jobs_row_level_ttl_total_rows{node_id="1",relation="defaultdb_public_events__events_pkey_"} 0 jobs_row_level_ttl_total_rows{node_id="1",relation="defaultdb_public_events_using_date__events_using_date_pkey_"} 0 ``` Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
`jobs.row_level_ttl.delete_duration`Duration for delete requests during row level TTL.HISTOGRAMNANOSECONDS
`jobs.row_level_ttl.num_active_spans`Number of active spans the TTL job is deleting from.GAUGECOUNT
`jobs.row_level_ttl.rows_deleted`Number of rows deleted by the row level TTL job.COUNTERCOUNT
`jobs.row_level_ttl.rows_selected`Number of rows selected for deletion by the row level TTL job.COUNTERCOUNT
`jobs.row_level_ttl.select_duration`Duration for select requests during row level TTL.HISTOGRAMNANOSECONDS
`jobs.row_level_ttl.span_total_duration`Duration for processing a span during row level TTL.HISTOGRAMNANOSECONDS
`jobs.row_level_ttl.total_expired_rows`Approximate number of rows that have expired the TTL on the TTL table.GAUGECOUNT
`jobs.row_level_ttl.total_rows`Approximate number of rows on the TTL table.GAUGECOUNT
### Metrics of type histogram When child metrics is enabled with changefeeds or row-level TTL jobs, be aware that metrics of type HISTOGRAM will increase cardinality quickly. For example, when you create two changefeeds with the metrics labels `employees` and `office_dogs`, the histogram metric `changefeed_flush_hist_nanos` exports child metrics for each bucket for each metrics label. In addition, the `default` scope will also be exported which includes changefeeds started without a metrics label. Therefore, in this example, `changefeed_flush_hist_nanos` exports child metrics for each bucket for the scope values: `default`, `employees` and `office_dogs`: ``` # HELP changefeed_flush_hist_nanos Time spent flushing messages across all changefeeds # TYPE changefeed_flush_hist_nanos histogram changefeed_flush_hist_nanos_bucket{node_id="1",le="5e+08"} 1 changefeed_flush_hist_nanos_bucket{node_id="1",le="5.572592853587432e+08"} 1 changefeed_flush_hist_nanos_bucket{node_id="1",le="6.210758222370743e+08"} 1 changefeed_flush_hist_nanos_bucket{node_id="1",le="6.922005377068515e+08"} 1 changefeed_flush_hist_nanos_bucket{node_id="1",le="7.714703539349157e+08"} 1 changefeed_flush_hist_nanos_bucket{node_id="1",le="8.598180362184556e+08"} 1 changefeed_flush_hist_nanos_bucket{node_id="1",le="9.582831688033088e+08"} 1 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.0680243876372875e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.1903330140009282e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.326648249442152e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.4785741108131227e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.6478983046833196e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.8366132632234223e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="2.046939589088547e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="2.2813521851760063e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="2.5426093767255764e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="2.833785368441068e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="3.1583064185550647e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="3.5199911554958534e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="3.9230955115614314e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="4.372362802333632e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="4.873079541115184e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="5.431137645156319e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="6.053103765649553e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="6.746296557296375e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="7.518872796674253e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="8.37992336275598e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="9.339580208980864e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.0409135585614676e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.1601174915283792e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.2929724885225649e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.4410418498852003e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.6060679028781363e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.789992503590971e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.9949798866972237e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="2.2234421319319225e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="2.478067546953807e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="2.7618523005723442e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="3.0781356785666904e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="3.4306393769506477e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="3.823511295046164e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="4.261374343677016e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="4.749380842807073e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="5.2932731487183495e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="5.899451224126824e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="6.575047946331352e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="7.328013039544164e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="8.167206619031862e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="9.102503447797786e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.0144909132590578e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.1306689626513626e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.2601515562088194e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.4044623113132697e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.5652993278314426e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.7445551695997415e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="1.9443391341601053e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="2.167002072794196e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="2.415164052912417e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="2.691745188300199e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="3.0000000000000085e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",le="+Inf"} 2 changefeed_flush_hist_nanos_sum{node_id="1"} 9.79696709e+08 changefeed_flush_hist_nanos_count{node_id="1"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="5e+08"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="5.572592853587432e+08"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="6.210758222370743e+08"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="6.922005377068515e+08"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="7.714703539349157e+08"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="8.598180362184556e+08"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="9.582831688033088e+08"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.0680243876372875e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.1903330140009282e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.326648249442152e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.4785741108131227e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.6478983046833196e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.8366132632234223e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.046939589088547e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.2813521851760063e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.5426093767255764e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.833785368441068e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.1583064185550647e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.5199911554958534e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.9230955115614314e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="4.372362802333632e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="4.873079541115184e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="5.431137645156319e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="6.053103765649553e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="6.746296557296375e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="7.518872796674253e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="8.37992336275598e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="9.339580208980864e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.0409135585614676e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.1601174915283792e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.2929724885225649e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.4410418498852003e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.6060679028781363e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.789992503590971e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.9949798866972237e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.2234421319319225e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.478067546953807e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.7618523005723442e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.0781356785666904e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.4306393769506477e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.823511295046164e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="4.261374343677016e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="4.749380842807073e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="5.2932731487183495e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="5.899451224126824e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="6.575047946331352e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="7.328013039544164e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="8.167206619031862e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="9.102503447797786e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.0144909132590578e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.1306689626513626e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.2601515562088194e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.4044623113132697e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.5652993278314426e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.7445551695997415e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.9443391341601053e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.167002072794196e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.415164052912417e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.691745188300199e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.0000000000000085e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="+Inf"} 0 changefeed_flush_hist_nanos_sum{node_id="1",scope="default"} 0 changefeed_flush_hist_nanos_count{node_id="1",scope="default"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="5e+08"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="5.572592853587432e+08"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="6.210758222370743e+08"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="6.922005377068515e+08"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="7.714703539349157e+08"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="8.598180362184556e+08"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="9.582831688033088e+08"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.0680243876372875e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.1903330140009282e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.326648249442152e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.4785741108131227e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.6478983046833196e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.8366132632234223e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.046939589088547e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.2813521851760063e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.5426093767255764e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.833785368441068e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.1583064185550647e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.5199911554958534e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.9230955115614314e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="4.372362802333632e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="4.873079541115184e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="5.431137645156319e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="6.053103765649553e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="6.746296557296375e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="7.518872796674253e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="8.37992336275598e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="9.339580208980864e+09"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.0409135585614676e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.1601174915283792e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.2929724885225649e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.4410418498852003e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.6060679028781363e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.789992503590971e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.9949798866972237e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.2234421319319225e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.478067546953807e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.7618523005723442e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.0781356785666904e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.4306393769506477e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.823511295046164e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="4.261374343677016e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="4.749380842807073e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="5.2932731487183495e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="5.899451224126824e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="6.575047946331352e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="7.328013039544164e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="8.167206619031862e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="9.102503447797786e+10"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.0144909132590578e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.1306689626513626e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.2601515562088194e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.4044623113132697e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.5652993278314426e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.7445551695997415e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.9443391341601053e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.167002072794196e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.415164052912417e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.691745188300199e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.0000000000000085e+11"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="+Inf"} 0 changefeed_flush_hist_nanos_sum{node_id="1",scope="employees"} 0 changefeed_flush_hist_nanos_count{node_id="1",scope="employees"} 0 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="5e+08"} 1 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="5.572592853587432e+08"} 1 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="6.210758222370743e+08"} 1 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="6.922005377068515e+08"} 1 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="7.714703539349157e+08"} 1 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="8.598180362184556e+08"} 1 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="9.582831688033088e+08"} 1 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.0680243876372875e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.1903330140009282e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.326648249442152e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.4785741108131227e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.6478983046833196e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.8366132632234223e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.046939589088547e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.2813521851760063e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.5426093767255764e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.833785368441068e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.1583064185550647e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.5199911554958534e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.9230955115614314e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="4.372362802333632e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="4.873079541115184e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="5.431137645156319e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="6.053103765649553e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="6.746296557296375e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="7.518872796674253e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="8.37992336275598e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="9.339580208980864e+09"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.0409135585614676e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.1601174915283792e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.2929724885225649e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.4410418498852003e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.6060679028781363e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.789992503590971e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.9949798866972237e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.2234421319319225e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.478067546953807e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.7618523005723442e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.0781356785666904e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.4306393769506477e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.823511295046164e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="4.261374343677016e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="4.749380842807073e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="5.2932731487183495e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="5.899451224126824e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="6.575047946331352e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="7.328013039544164e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="8.167206619031862e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="9.102503447797786e+10"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.0144909132590578e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.1306689626513626e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.2601515562088194e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.4044623113132697e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.5652993278314426e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.7445551695997415e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.9443391341601053e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.167002072794196e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.415164052912417e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.691745188300199e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.0000000000000085e+11"} 2 changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="+Inf"} 2 changefeed_flush_hist_nanos_sum{node_id="1",scope="office_dogs"} 9.79696709e+08 changefeed_flush_hist_nanos_count{node_id="1",scope="office_dogs"} 2 ``` ## Enable `database` and `application_name` labels > [!NOTE] > > **This feature is in [preview](/docs/releases/cockroachdb-feature-availability.md)** and subject to change. To share feedback and/or issues, contact [Support](https://support.cockroachlabs.com). The following cluster settings enable the `database` and `application_name` labels for certain metrics, along with their internal counterparts if they exist: * [`sql.metrics.database_name.enabled`](/docs/stable/cluster-settings#setting-sql-metrics-database-name-enabled) * [`sql.metrics.application_name.enabled`](/docs/stable/cluster-settings#setting-sql-metrics-application-name-enabled) By default, these cluster settings are disabled. To enable them, use the [`SET CLUSTER SETTING`](/docs/stable/set-cluster-setting) statement. Because these labels use aggregate metrics, you must enable the [`server.child_metrics.enabled`](#enable-child-metrics) cluster setting to use them. ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} SET CLUSTER SETTING server.child_metrics.enabled = true; ``` ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} SET CLUSTER SETTING sql.metrics.database_name.enabled = true; ``` ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} SET CLUSTER SETTING sql.metrics.application_name.enabled = true; ``` Toggling the `sql.metrics.database_name.enabled` and `sql.metrics.application_name.enabled` cluster settings clears existing metric values for current label combinations and reinitializes the affected metrics to reflect the new label configuration. When toggling the `sql.metrics.database_name.enabled` and `sql.metrics.application_name.enabled` cluster settings, only the values for existing metric label combinations will be cleared. Aggregated metric values for the affected metrics will not be cleared. > [!NOTE] > > Child metrics (metrics with the `database` and `application_name` labels) are independent from the parent (aggregated metric). The child metrics are initialized when the cluster settings are enabled. > > For this reason, child `COUNTER` metrics may not always add up to the parent `COUNTER` metric. For an example, refer to [Examples 1 through 6](#1-all-cluster-settings-disabled). > > For `GAUGE` metrics, values may be different and potentially unexpected depending on when a setting is enabled. For an example, refer to [7. `GAUGE` metric example](#7-gauge-metric-example). These labels affect only the metrics emitted via [Prometheus export](/docs/stable/prometheus-endpoint.md). They are not visible in the [DB Console Metrics dashboards](/docs/stable/monitoring-and-alerting.md#metrics-dashboards). The system retains up to 5,000 recently used label combinations. Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescription When AggregatedTypeUnit
`sql.crud_query.count`Number of SQL SELECT, INSERT, UPDATE, DELETE statements successfully executedCOUNTERCOUNT
`sql.crud_query.count.internal`Number of SQL SELECT, INSERT, UPDATE, DELETE statements successfully executed (internal queries)COUNTERCOUNT
`sql.delete.count`Number of SQL DELETE statements successfully executedCOUNTERCOUNT
`sql.delete.count.internal`Number of SQL DELETE statements successfully executed (internal queries)COUNTERCOUNT
`sql.distsql.contended_queries.count`Number of SQL queries that experienced contentionCOUNTERCOUNT
`sql.distsql.cumulative_contention_nanos`Cumulative contention across all queries (in nanoseconds)COUNTERNANOSECONDS
`sql.failure.count`Number of statements resulting in a planning or runtime errorCOUNTERCOUNT
`sql.failure.count.internal`Number of statements resulting in a planning or runtime error (internal queries)COUNTERCOUNT
`sql.full.scan.count`Number of full table or index scansCOUNTERCOUNT
`sql.full.scan.count.internal`Number of full table or index scans (internal queries)COUNTERCOUNT
`sql.insert.count`Number of SQL INSERT statements successfully executedCOUNTERCOUNT
`sql.insert.count.internal`Number of SQL INSERT statements successfully executed (internal queries)COUNTERCOUNT
`sql.select.count`Number of SQL SELECT statements successfully executedCOUNTERCOUNT
`sql.select.count.internal`Number of SQL SELECT statements successfully executed (internal queries)COUNTERCOUNT
`sql.service.latency`Latency of SQL request executionHISTOGRAMNANOSECONDS
`sql.service.latency.internal`Latency of SQL request execution (internal queries)HISTOGRAMNANOSECONDS
`sql.statements.active`Number of currently active user SQL statementsGAUGECOUNT
`sql.statements.active.internal`Number of currently active user SQL statements (internal queries)GAUGECOUNT
`sql.txn.begin.count`Number of SQL transaction BEGIN statements successfully executedCOUNTERCOUNT
`sql.txn.begin.count.internal`Number of SQL transaction BEGIN statements successfully executed (internal queries)COUNTERCOUNT
`sql.txn.commit.count`Number of SQL transaction COMMIT statements successfully executedCOUNTERCOUNT
`sql.txn.commit.count.internal`Number of SQL transaction COMMIT statements successfully executed (internal queries)COUNTERCOUNT
`sql.txn.latency`Latency of SQL transactionsHISTOGRAMNANOSECONDS
`sql.txn.latency.internal`Latency of SQL transactions (internal queries)HISTOGRAMNANOSECONDS
`sql.txn.rollback.count`Number of SQL transaction ROLLBACK statements successfully executedCOUNTERCOUNT
`sql.txn.rollback.count.internal`Number of SQL transaction ROLLBACK statements successfully executed (internal queries)COUNTERCOUNT
`sql.txns.open`Number of currently open user SQL transactionsGAUGECOUNT
`sql.txns.open.internal`Number of currently open user SQL transactions (internal queries)GAUGECOUNT
`sql.update.count`Number of SQL UPDATE statements successfully executedCOUNTERCOUNT
`sql.update.count.internal`Number of SQL UPDATE statements successfully executed (internal queries)COUNTERCOUNT
### Examples This section demonstrates the impact of enabling and disabling the relevant cluster settings. [Examples 1 through 6](#1-all-cluster-settings-disabled) show the effect on the `COUNTER` metric `sql.select.count`. During these examples, the [`movr` workload](/docs/stable/cockroach-workload#run-the-movr-workload) was running on node 1. The aggregated metric consistently increases as the examples progress. [Example 7](#7-gauge-metric-example) shows a possible effect on the `GAUGE` metric `sql.txn.open`. #### 1. All cluster settings disabled ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} SET CLUSTER SETTING server.child_metrics.enabled = false; SET CLUSTER SETTING sql.metrics.database_name.enabled = false; SET CLUSTER SETTING sql.metrics.application_name.enabled = false; ``` The Prometheus export only gives the aggregated metric for the node. ``` # HELP sql_select_count Number of SQL SELECT statements successfully executed # TYPE sql_select_count counter sql_select_count{node_id="1"} 2030 ``` #### 2. Only child metrics enabled ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} SET CLUSTER SETTING server.child_metrics.enabled = true; SET CLUSTER SETTING sql.metrics.database_name.enabled = false; SET CLUSTER SETTING sql.metrics.application_name.enabled = false; ``` The Prometheus export still only gives the aggregated metric for the node. ``` sql_select_count{node_id="1"} 6568 ``` #### 3. Child metrics and `database_name` label enabled ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} SET CLUSTER SETTING server.child_metrics.enabled = true; SET CLUSTER SETTING sql.metrics.database_name.enabled = true; SET CLUSTER SETTING sql.metrics.application_name.enabled = false; ``` The aggregated metric and a child metric with only the `database` label are emitted. ``` sql_select_count{node_id="1"} 10259 sql_select_count{node_id="1",database="movr"} 816 ``` #### 4. Child metrics and `application_name` label enabled ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} SET CLUSTER SETTING server.child_metrics.enabled = true; SET CLUSTER SETTING sql.metrics.database_name.enabled = false; SET CLUSTER SETTING sql.metrics.application_name.enabled = true; ``` The aggregated metric and a child metric with only the `application_name` label are emitted. Note that even though the aggregated metric has increased, the child metric with `application_name` label has a value less than the child metric with `database` label in the [preceding example](#3-child-metrics-and-database_name-label-enabled). This is because the labeled metrics have been reset, while the aggregated metric was not reset. ``` sql_select_count{node_id="1"} 14077 sql_select_count{node_id="1",application_name="movr"} 718 ``` #### 5. Child metrics and both `database` and `application_name` label enabled ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} SET CLUSTER SETTING server.child_metrics.enabled = true; SET CLUSTER SETTING sql.metrics.database_name.enabled = true; SET CLUSTER SETTING sql.metrics.application_name.enabled = true; ``` The aggregated metric and a child metric with both `database` and `application_name` labels are emitted. ``` sql_select_count{node_id="1"} 21085 sql_select_count{node_id="1",database="movr",application_name="movr"} 3962 ``` #### 6. Aggregate metric disabled The [cluster setting `server.child_metrics.include_aggregate.enabled`](/docs/stable/cluster-settings#setting-server-child-metrics-include-aggregate-enabled) (default: `true`) reports an aggregate time series for applicable multi-dimensional metrics. When set to `false`, it stops reporting the aggregate time series, preventing double counting when querying those metrics. ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} SET CLUSTER SETTING server.child_metrics.enabled = true; SET CLUSTER SETTING sql.metrics.database_name.enabled = true; SET CLUSTER SETTING sql.metrics.application_name.enabled = true; SET CLUSTER SETTING server.child_metrics.include_aggregate.enabled = false; ``` No aggregated metric emitted. Only the child metric with both the `database` and `application_name` labels is emitted. ``` sql_select_count{node_id="1",database="movr",application_name="movr"} 8703 ``` #### 7. `GAUGE` metric example Changes to cluster settings may take time to reinitialize affected metrics. As a result, some `GAUGE` metrics might briefly show unexpected values. `GAUGE` values for both aggregated and child metrics increase and decrease as transactions are opened and closed. This example uses the `GAUGE` metric `sql.txn.open`. Consider the following scenario: | Time | Action | `sql.txn.open` aggregated metric | `sql.txn.open` child metric | | :-: | - | :-: | :-: | | 1 | Open a transaction. The value of the `sql.txn.open` aggregated metric is incremented. | 1 | - | | 2 | Enable `sql.metrics.database_name.enabled` and `sql.metrics.application_name.enabled`. Re-initialize child metrics. | 1 | 0 | | 3 | Close a transaction. The values of both the aggregated and child `sql.txn.open` metrics are decremented. | 0 | -1 | To avoid negative values in child metrics, use the [Prometheus `clamp_min` function](https://prometheus.io/docs/prometheus/latest/querying/functions/#clamp_min) to set the metric to zero. ## Enable detailed latency metrics The [cluster setting `sql.stats.detailed_latency_metrics.enabled`](/docs/stable/cluster-settings#setting-sql-stats-detailed-latency-metrics-enabled) labels the latency metric `sql.exec.latency.detail` with the [statement fingerprint](/docs/stable/ui-statements-page.md#sql-statement-fingerprints). To estimate the cardinality of the set of all statement fingerprints, use the `sql.query.unique.count` metric. For most workloads, this metric ranges from dozens to hundreds. `sql.stats.detailed_latency_metrics.enabled` is disabled by default. To enable it, use the [`SET CLUSTER SETTING`](/docs/stable/set-cluster-setting) statement. ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} SET CLUSTER SETTING sql.stats.detailed_latency_metrics.enabled = true; ``` > [!CAUTION] > > Be aware that the `sql.exec.latency.detail` metric is of type HISTOGRAM and will quickly increase the cardinality of exported metrics. For workloads with over a couple thousand statement fingerprints, exercise caution when enabling `sql.stats.detailed_latency_metrics.enabled`. For workloads with tens of thousands of distinct query fingerprints, leave this cluster setting set to `false`. Following is a list of the metrics that have multi-dimensional metrics:
CockroachDB Metric NameDescriptionTypeUnit
`sql.exec.latency.detail`Latency of SQL statement execution, by statement fingerprintHISTOGRAMNANOSECONDS
For example: ``` # HELP sql_query_unique_count Cardinality estimate of the set of statement fingerprints # TYPE sql_query_unique_count counter sql_query_unique_count{node_id="1"} 48 ... # HELP sql_exec_latency_detail Latency of SQL statement execution, by statement fingerprint # TYPE sql_exec_latency_detail histogram sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="10000"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="12638.482029342978"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="15973.122800602541"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="20187.602546790382"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="25514.065200312878"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="32245.90545296394"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="40753.92965871775"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="51506.78076168121"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="65096.75230458167"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="82272.41341700466"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="103979.84184814895"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="131414.73626117557"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="166088.27826277143"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="209910.3720108553"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="265294.8464431894"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="335292.41492495546"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="423758.7160604059"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="535566.6917706894"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="676875.0009458527"} 0 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="855467.2535565672"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.0811807510766068e+06"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.3664483492953242e+06"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.7269832906594332e+06"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="2.182644728397485e+06"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="2.7585316176291797e+06"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="3.4863652276780806e+06"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="4.406236427773566e+06"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="5.568813990945262e+06"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="7.038135554931545e+06"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="8.89513497310822e+06"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.1242100350620849e+07"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.4208308325339198e+07"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.795714494371637e+07"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="2.269510536694665e+07"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="2.868316813342006e+07"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="3.625117049988527e+07"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="4.581597669054482e+07"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="5.790443980602476e+07"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="7.318242219076161e+07"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="9.249147277217315e+07"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.1689518164985757e+08"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.4773776525985083e+08"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.8671810912919158e+08"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="2.359833466782189e+08"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="2.9824712862168837e+08"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="3.769390975388353e+08"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="4.7639380104013294e+08"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="6.020894493336115e+08"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="7.609496685459859e+08"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="9.61724871115294e+08"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.2154742500762835e+09"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.5361749466718242e+09"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="1.941491945743876e+09"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="2.453751106639811e+09"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="3.10116892657477e+09"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="3.919406774847209e+09"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="4.953535208959157e+09"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="6.260516572014802e+09"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="7.912342618981298e+09"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="9.99999999999997e+09"} 1 sql_exec_latency_detail_bucket{fingerprint="SHOW database",node_id="1",le="+Inf"} 1 sql_exec_latency_detail_sum{fingerprint="SHOW database",node_id="1"} 723750 sql_exec_latency_detail_count{fingerprint="SHOW database",node_id="1"} 1 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="10000"} 0 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="12638.482029342978"} 0 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="15973.122800602541"} 0 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="20187.602546790382"} 0 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="25514.065200312878"} 0 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="32245.90545296394"} 0 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="40753.92965871775"} 55 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="51506.78076168121"} 319 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="65096.75230458167"} 801 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="82272.41341700466"} 1503 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="103979.84184814895"} 2307 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="131414.73626117557"} 2975 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="166088.27826277143"} 3439 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="209910.3720108553"} 3779 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="265294.8464431894"} 3912 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="335292.41492495546"} 4039 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="423758.7160604059"} 4156 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="535566.6917706894"} 4236 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="676875.0009458527"} 4293 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="855467.2535565672"} 4308 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.0811807510766068e+06"} 4323 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.3664483492953242e+06"} 4330 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.7269832906594332e+06"} 4343 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="2.182644728397485e+06"} 4347 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="2.7585316176291797e+06"} 4348 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="3.4863652276780806e+06"} 4349 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="4.406236427773566e+06"} 4349 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="5.568813990945262e+06"} 4349 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="7.038135554931545e+06"} 4349 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="8.89513497310822e+06"} 4349 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.1242100350620849e+07"} 4350 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.4208308325339198e+07"} 4350 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.795714494371637e+07"} 4350 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="2.269510536694665e+07"} 4351 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="2.868316813342006e+07"} 4354 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="3.625117049988527e+07"} 4359 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="4.581597669054482e+07"} 4363 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="5.790443980602476e+07"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="7.318242219076161e+07"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="9.249147277217315e+07"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.1689518164985757e+08"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.4773776525985083e+08"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.8671810912919158e+08"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="2.359833466782189e+08"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="2.9824712862168837e+08"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="3.769390975388353e+08"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="4.7639380104013294e+08"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="6.020894493336115e+08"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="7.609496685459859e+08"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="9.61724871115294e+08"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.2154742500762835e+09"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.5361749466718242e+09"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="1.941491945743876e+09"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="2.453751106639811e+09"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="3.10116892657477e+09"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="3.919406774847209e+09"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="4.953535208959157e+09"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="6.260516572014802e+09"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="7.912342618981298e+09"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="9.99999999999997e+09"} 4367 sql_exec_latency_detail_bucket{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1",le="+Inf"} 4367 sql_exec_latency_detail_sum{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1"} 1.27599889e+09 sql_exec_latency_detail_count{fingerprint="SELECT city, id FROM vehicles WHERE city = _",node_id="1"} 4367 ```