> ## Documentation Index > Fetch the complete documentation index at: https://cockroachlabs.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Cluster Monitoring export const InternalLink = ({version, path = "", children, ...props}) => { let detectedVersion = version || "stable"; if (typeof window !== 'undefined' && !version) { const match = window.location.pathname.match(/\/docs\/([^/]+)/); if (match) { detectedVersion = match[1]; } } const normalizedPath = path.startsWith("/") ? path.slice(1) : path; return {children} ; }; This article assumes you have already deployed CockroachDB on a single Kubernetes cluster. Despite CockroachDB's various built-in safeguards against failure, it is critical to actively monitor the overall health and performance of a cluster running in production and to create alerting rules that promptly send notifications when there are events that require investigation or intervention. All `kubectl` steps should be performed in the namespace where you installed the Operator. By default, this is `cockroach-operator-system`. If you deployed CockroachDB on Red Hat OpenShift, substitute `kubectl` with `oc` in the following commands. ## Configure Prometheus Every node of a CockroachDB cluster exports granular timeseries metrics formatted for easy integration with [Prometheus](https://prometheus.io/), a tool for storing, aggregating, and querying timeseries data. This section shows you how to orchestrate Prometheus as part of your Kubernetes cluster and pull these metrics into Prometheus for external monitoring. This guidance is based on [CoreOS's Prometheus Operator](https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md), which allows a Prometheus instance to be managed using built-in Kubernetes concepts. If you're on Hosted GKE, before starting, make sure the email address associated with your Google Cloud account is part of the `cluster-admin` RBAC group, as shown in Deploy CockroachDB with Kubernetes. 1. From your local workstation, edit the `cockroachdb` service to add the `prometheus: cockroachdb` label: ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ kubectl label svc cockroachdb prometheus=cockroachdb ``` ``` service/cockroachdb labeled ``` This ensures that only the `cockroachdb` (not the `cockroach-public` service) is being monitored by a Prometheus job. ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ kubectl label svc cockroachdb prometheus=cockroachdb ``` ``` service/cockroachdb labeled ``` This ensures that only the `cockroachdb` (not the `cockroach-public` service) is being monitored by a Prometheus job. ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ kubectl label svc my-release-cockroachdb prometheus=cockroachdb ``` ``` service/my-release-cockroachdb labeled ``` This ensures that there is a Prometheus job and monitoring data only for the `my-release-cockroachdb` service, not for the `my-release-cockroach-public` service. 2. Determine the latest version of [CoreOS's Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator/releases/) and run the following to download and apply the latest `bundle.yaml` definition file: Be sure to specify the latest [CoreOS Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator/releases/) version in the following command, in place of this example's use of version `v0.58.0`. ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ kubectl apply \ -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.58.0/bundle.yaml \ --server-side ``` ``` customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com serverside-applied customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com serverside-applied customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com serverside-applied customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com serverside-applied customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com serverside-applied customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com serverside-applied customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com serverside-applied clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator serverside-applied clusterrole.rbac.authorization.k8s.io/prometheus-operator serverside-applied deployment.apps/prometheus-operator serverside-applied serviceaccount/prometheus-operator serverside-applied service/prometheus-operator serverside-applied ``` 1. Confirm that the `prometheus-operator` has started: ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ kubectl get deploy prometheus-operator ``` ``` NAME READY UP-TO-DATE AVAILABLE AGE prometheus-operator 1/1 1 1 27s ``` 2. Download our Prometheus manifest: ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ curl -O https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/prometheus/prometheus.yaml ``` By default, this manifest uses the secret name generated by the CockroachDB Kubernetes Operator. If you generated your own certificates and keys when starting CockroachDB, be sure that `ca.secret.name` matches the name of the node secret you created. 1. Apply the Prometheus manifest. This creates the various objects necessary to run a Prometheus instance: ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ kubectl apply -f prometheus.yaml ``` ``` serviceaccount/prometheus created clusterrole.rbac.authorization.k8s.io/prometheus created clusterrolebinding.rbac.authorization.k8s.io/prometheus created servicemonitor.monitoring.coreos.com/cockroachdb created prometheus.monitoring.coreos.com/cockroachdb created ``` 2. Access the Prometheus UI locally and verify that CockroachDB is feeding data into Prometheus: 1. Port-forward from your local machine to the pod running Prometheus: ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ kubectl port-forward prometheus-cockroachdb-0 9090 ``` 2. Go to [http://localhost:9090](http://localhost:9090) in your browser. 3. To verify that each CockroachDB node is connected to Prometheus, go to **Status > Targets**. The screen should look like this: Prometheus targets

4. To verify that data is being collected, go to **Graph**, enter the `sys_uptime` variable in the field, click **Execute**, and then click the **Graph** tab. The screen should like this: Prometheus graph

Prometheus auto-completes CockroachDB time series metrics for you, but if you want to see a full listing, with descriptions, port-forward as described in Access the DB ConsoleAccess the DB Console and then point your browser to [http://localhost:8080/\_status/vars](http://localhost:8080/_status/vars). For more details on using the Prometheus UI, see their [official documentation](https://prometheus.io/docs/introduction/getting_started/). ## Configure Alertmanager Active monitoring helps you spot problems early, but it is also essential to send notifications when there are events that require investigation or intervention. This section shows you how to use [Alertmanager](https://prometheus.io/docs/alerting/alertmanager/) and CockroachDB's starter alerting rules to do this. 1. Download our `alertmanager-config.yaml` configuration file: ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ curl -O \ https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/prometheus/alertmanager-config.yaml ``` 2. Edit the `alertmanager-config.yaml` file to [specify the desired receivers for notifications](https://prometheus.io/docs/alerting/configuration/#receiver). Initially, the file contains a placeholder web hook. 3. Add this configuration to the Kubernetes cluster as a secret, renaming it to `alertmanager.yaml` and labelling it to make it easier to find: ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ kubectl create secret generic alertmanager-cockroachdb \ --from-file=alertmanager.yaml=alertmanager-config.yaml ``` ``` secret/alertmanager-cockroachdb created ``` ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ kubectl label secret alertmanager-cockroachdb app=cockroachdb ``` ``` secret/alertmanager-cockroachdb labeled ``` The name of the secret, `alertmanager-cockroachdb`, must match the name used in the `alertmanager.yaml` file. If they differ, the Alertmanager instance will start without configuration, and nothing will happen. 1. Use our `alertmanager.yaml` file to create the various objects necessary to run an Alertmanager instance, including a ClusterIP service so that Prometheus can forward alerts: ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ kubectl apply \ -f https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/prometheus/alertmanager.yaml ``` ``` alertmanager.monitoring.coreos.com/cockroachdb created service/alertmanager-cockroachdb created ``` 2. Verify that Alertmanager is running: 1. Port-forward from your local machine to the pod running Alertmanager: ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ kubectl port-forward alertmanager-cockroachdb-0 9093 ``` 2. Go to [http://localhost:9093](http://localhost:9093) in your browser. The screen should look like this:

3. Ensure that the Alertmanagers are visible to Prometheus by opening [http://localhost:9090/status](http://localhost:9090/status). The screen should look like this:

4. Add CockroachDB's starter alerting rules: ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ kubectl apply \ -f https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/prometheus/alert-rules.yaml ``` ``` prometheusrule.monitoring.coreos.com/prometheus-cockroachdb-rules created ``` 5. Ensure that the rules are visible to Prometheus by opening [http://localhost:9090/rules](http://localhost:9090/rules). The screen should look like this:

6. Verify that the `TestAlertManager` example alert is firing by opening [http://localhost:9090/alerts](http://localhost:9090/alerts). The screen should look like this:

7. To remove the example alert: 1. Use the `kubectl edit` command to open the rules for editing: ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ kubectl edit prometheusrules prometheus-cockroachdb-rules ``` 2. Remove the `dummy.rules` block and save the file: ``` - name: rules/dummy.rules rules: - alert: TestAlertManager expr: vector(1) ``` ## Configure logging When running CockroachDB v21.1 and later, you can use the Operator to configure the CockroachDB logging system. This allows you to output logs to configurable log sinks such as file or network logging destinations. By default, Kubernetes deployments running CockroachDB v20.2 or earlier output all logs to `stderr`. The logging configuration is defined in a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) object, using a key named `logging.yaml`. For example: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} apiVersion: v1 data: logging.yaml: | sinks: file-groups: dev: channels: DEV filter: WARNING fluent-servers: ops: channels: [OPS, HEALTH, SQL_SCHEMA] address: 127.0.0.1:5170 net: tcp redact: true security: channels: [SESSIONS, USER_ADMIN, PRIVILEGES, SENSITIVE_ACCESS] address: 127.0.0.1:5170 net: tcp auditable: true kind: ConfigMap metadata: name: logconfig namespace: cockroach-operator-system ``` The above configuration overrides the default logging configuration and reflects our recommended Kubernetes logging configuration: * Save debug-level logs (the `DEV` log channel) to disk for troubleshooting. * Send operational- and security-level logs to a network collector, in this case Fluentd. The ConfigMap `name` must match the `logConfigMap` object of the Operator's custom resource, which is used to deploy the cluster: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} spec: logConfigMap: logconfig ``` By default, the Operator also modifies the default logging configuration with the following: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} sinks: stderr: channels: OPS redact: true ``` This outputs logging events in the `OPS` channel to a `cockroach-stderr.log` file. ### Example: Creating a troubleshooting log file on pods In this example, CockroachDB has already been deployed on a Kubernetes cluster. We override the default logging configuration to output `DEV` logs to a `cockroach-dev.log` file. 1. Create a ConfigMap named `logconfig`. Note that `namespace` is set to the Operator's default namespace (`cockroach-operator-system`): ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} apiVersion: v1 data: logging.yaml: | sinks: file-groups: dev: channels: DEV filter: WARNING kind: ConfigMap metadata: name: logconfig namespace: cockroach-operator-system ``` For simplicity, also name the YAML file `logconfig.yaml`. The ConfigMap key is not related to the ConfigMap `name` or YAML filename, and **must** be named `logging.yaml`. This configuration outputs `DEV` logs that have severity `WARNING` to a `cockroach-dev.log` file on each pod. 1. Apply the ConfigMap to the cluster: ``` kubectl apply -f logconfig.yaml ``` ``` configmap/logconfig created ``` 2. Add the `name` of the ConfigMap in `logConfigMap` to the Operator's custom resource: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} spec: logConfigMap: logconfig ``` 3. Apply the new settings to the cluster: ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ kubectl apply -f example.yaml ``` The changes will be rolled out to each pod. 4. See the log files available on a pod: ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ kubectl exec cockroachdb-2 -- ls cockroach-data/logs ``` ``` cockroach-dev.cockroachdb-2.unknownuser.2022-05-02T19_03_03Z.000001.log cockroach-dev.log cockroach-health.cockroachdb-2.unknownuser.2022-05-02T18_53_01Z.000001.log cockroach-health.log cockroach-pebble.cockroachdb-2.unknownuser.2022-05-02T18_52_48Z.000001.log cockroach-pebble.log cockroach-stderr.cockroachdb-2.unknownuser.2022-05-02T18_52_48Z.000001.log cockroach-stderr.cockroachdb-2.unknownuser.2022-05-02T19_03_03Z.000001.log cockroach-stderr.cockroachdb-2.unknownuser.2022-05-02T20_04_03Z.000001.log cockroach-stderr.log cockroach.cockroachdb-2.unknownuser.2022-05-02T18_52_48Z.000001.log cockroach.log ... ``` 5. View a specific log file: ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ kubectl exec cockroachdb-2 -- cat cockroach-data/logs/cockroach-dev.log ``` ## See also * Monitoring and Alerting * Metrics