> ## Documentation Index > Fetch the complete documentation index at: https://cockroachlabs.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Configure Logs export const InternalLink = ({version, path = "", children, ...props}) => { let detectedVersion = version || "stable"; if (typeof window !== 'undefined' && !version) { const match = window.location.pathname.match(/\/docs\/([^/]+)/); if (match) { detectedVersion = match[1]; } } const normalizedPath = path.startsWith("/") ? path.slice(1) : path; return {children} ; }; This page describes how to configure CockroachDB logs with the `--log` or `log-config-file` flag and a [YAML payload](#yaml-payload). Most logging behaviors are configurable, including: * The [log sinks](#configure-log-sinks) that output logs to different locations, including over the network. * The logging channels that are mapped to each sink. * The format used by the log messages. * The [redaction](#redact-logs) of log messages. * The [buffering](#log-buffering-for-network-sinks) of log messages. For examples of how these settings can be used in practice, see Logging Use Cases. ## Flag To configure the logging behavior of a `cockroach` command, include one of these flags with the command: * `--log={yaml}`, where {'{yaml}'} is the [YAML payload](#yaml-payload) * `--log-config-file={yaml-file}`, where {'{yaml-file}'} is the path to a YAML file To disable logging, set `--log-dir` to a blank directory (`--log-dir=`) instead of using one of the other logging flags. Do not use `--log-dir=""`; this creates a new directory named `""` and stores log files in that directory. All `cockroach` commands support logging and can be configured with `--log` or `--log-config-file`. However, note that most messages related to cluster operation are generated by `cockroach start` or `cockroach start-single-node`. Other commands generate messages related to their own execution, which are mainly useful when troubleshooting the behaviors of those commands. ## YAML payload All log settings for a `cockroach` command are specified with a YAML payload in one of the following formats: * Block format, where each parameter is written on a separate line. For example, after creating a file `logs.yaml`, pass the YAML values with either `--log-config-file` or `--log`: ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ cockroach start-single-node --certs-dir=certs --log-config-file=logs.yaml ``` ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ cockroach start-single-node --certs-dir=certs --log="$(cat logs.yaml)" ``` * Inline format, where all parameters are specified on one line. For example, to generate an `ops` log file that collects the `OPS` and `HEALTH` channels (overriding the file groups defined for those channels in the [default configuration](#default-logging-configuration)): ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} $ cockroach start-single-node --certs-dir=certs --log="sinks: {file-groups: {ops: {channels: [OPS, HEALTH]}}}" ``` Note that the inline spaces must be preserved. For clarity, this article uses the block format to describe the YAML payload, which has the overall structure: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} file-defaults: ... # defaults inherited by file sinks fluent-defaults: ... # defaults inherited by Fluentd sinks http-defaults: ... # defaults inherited by HTTP sinks sinks: file-groups: ... # file sink definitions fluent-servers: ... # Fluentd sink definitions http-servers: ... # HTTP sink definitions stderr: ... # stderr sink definitions capture-stray-errors: ... # parameters for the stray error capture system ``` Providing a logging configuration is optional. Any fields included in the YAML payload will override the same fields in the [default logging configuration](#default-logging-configuration). You can view the default settings by running `cockroach debug check-log-config`, which returns the YAML definitions and a URL to a visualization of the default logging configuration. ## Configure log sinks Log *sinks* route events from specified logging channels to destinations outside CockroachDB. These destinations currently include [log files](#output-to-files), [Fluentd](https://www.fluentd.org/)-compatible [servers](#output-to-fluentd-compatible-network-collectors), [HTTP servers](#output-to-http-network-collectors), and the [standard error stream (`stderr`)](#output-to-stderr). All supported output destinations are configured under `sinks`: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} file-defaults: ... fluent-defaults: ... http-defaults: ... sinks: file-groups: {file group name}: channels: {channels} ... fluent-servers: {server name}: channels: {channels} ... http-servers: {server name}: channels: {channels} ... stderr: channels: {channels} ... ``` All supported sink types use the following common sink parameters: | Parameter | Description | | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `filter` | Minimum severity level at which logs enter the channels selected for the sink. Accepts one of the valid severity levels or `NONE`, which excludes all messages from the sink output. For details, see [Set logging levels](#set-logging-levels). | | | `format` | Log message format to use for file or network sinks. Accepts one of the valid log formats. For details, see [file logging format](#file-logging-format), [Fluentd logging format](#fluentd-logging-format), and [HTTP logging format](#http-logging-format). | | | `format-options` | Customization options for specified `format`. For available options for each format, see Log formats. For an example use case, see [Set timezone](#set-timezone). | | | `redact` | When `true`, enables automatic redaction of personally identifiable information (PII) from log messages. This ensures that sensitive data is not transmitted when collecting logs centrally or over a network. For details, see [Redact logs](#redact-logs). | | | `redactable` | When `true`, preserves redaction markers around fields that are considered sensitive in the log messages. The markers are recognized by `cockroach debug zip` and `cockroach debug merge-logs` but may not be compatible with external log collectors. For details on how the markers appear in each format, see Log formats. | | | `exit-on-error` | When `true`, stops the Cockroach node if an error is encountered while writing to the sink. We recommend enabling this option on file sinks in order to avoid losing any log entries. When set to `false`, this can be used to mark certain sinks (such as `stderr`) as non-critical. | | | | `auditable` | If `true`, enables `exit-on-error` on the sink. Also disables `buffered-writes` if the sink is under `file-groups`. This guarantees [non-repudiability](https://wikipedia.org/wiki/Non-repudiation) for any logs in the sink, but can incur a performance overhead and higher disk IOPS consumption. This setting is typically enabled for security-related logs. File-based audit logging cannot coexist with the buffering configuration, so disable either [`buffering`](#file-buffering) or `auditable`. | If not specified for a given sink, these parameter values are inherited from [`file-defaults`](#set-file-defaults) (for file sinks), [`fluent-defaults`](#set-fluentd-defaults) (for Fluentd sinks), and [`http-defaults`](#set-http-defaults) (for HTTP sinks). ### Output to files CockroachDB can write messages to one or more log files. `file-groups` specifies the channels that output to each log file, along with its output directory and other configuration details. For example: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} file-defaults: ... fluent-defaults: ... http-defaults: ... sinks: file-groups: default: channels: [DEV] health: channels: [HEALTH] dir: health-logs ... ``` A file group name is arbitrary and is used to name the log files. The `default` file group is an exception. For details, see [Log file naming](#log-file-naming). Along with the [common sink parameters](#common-sink-parameters), each file group accepts the following additional parameters: | Parameter | Description | | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `channels` | List of channels that output to this sink. Use a YAML array or string of channel names, `ALL` to include all channels, or `ALL EXCEPT {channels}` to include all channels except the specified channel names. For more details on acceptable syntax, see [Logging channel selection](#logging-channel-selection). | | `dir` | Output directory for log files generated by this sink. | | `max-file-size` | Approximate maximum size of individual files generated by this sink. Used to specify the maximum size that a log file can grow before a new log file is created. Accepts a valid file size, such as `2MiB`. **Default:**`10MiB` | | `max-group-size` | Approximate maximum combined size of all files to be preserved for this sink. Configures the maximum size for a logging group (for example, `cockroach`, `cockroach-sql-audit`, `cockroach-auth`, `cockroach-sql-exec`, `cockroach-pebble`), after which the oldest log file is deleted. An asynchronous garbage collection removes files that cause the file set to grow beyond this specified size. Accepts a valid file size, such as `1GiB`. For high-traffic deployments, or to ensure log retention over longer periods of time, consider raising this value to `500MiB` or `1GiB`. **Default:**`100MiB` | | `file-permissions` | The `chmod`-style permissions on generated log files, formatted as a 3-digit octal number. The executable bit must not be set. **Default:**`640` (readable by the owner or members of the group, writable by the owner). | | `buffered-writes` | When `true`, enables buffering of writes. Set to `false` to flush every log entry (i.e., propagate data from the `cockroach` process to the OS) and synchronize writes (i.e., ask the OS to confirm the log data was written to disk). Disabling this setting provides [non-repudiation](https://wikipedia.org/wiki/Non-repudiation) guarantees, but can incur a performance overhead and higher disk IOPS consumption. This setting is typically disabled for security-related logs. | | `buffering` | `buffering` is disabled by default for file log sinks. Default: `buffering: NONE`. Note that enabling asynchronous buffering of file log sinks is in preview.

To configure buffering of log messages for the sink, use the following sub-parameters:When `max-staleness` and `flush-trigger-size` are used together, whichever is reached first will trigger the flush. This setting is typically disabled for security-related logs. For a usage example, refer to Enable WAL failover.

File-based audit logging cannot coexist with this buffering configuration, so disable either `buffering` or [`auditable`](#auditable). | If not specified for a given file group, the parameter values are inherited from [`file-defaults`](#configure-logging-defaults) (except `channels`, which uses the [default configuration](#default-logging-configuration)). #### Log file naming Log files are named in the following format: ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} {process}-{file group}.{host}.{user}.{start timestamp in UTC}.{process ID}.log ``` For example, a file group `health` will generate a log file that looks like this: ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} cockroach-health.work-laptop.worker.2021-03-15T15_24_10Z.024338.log ``` For each file group, a symlink points to the latest generated log file. It's easiest to refer to the symlink. For example: ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} cockroach-health.log ``` The files generated for a group named `default` are named after the pattern `cockroach.{metadata}.log`. #### Access in DB Console Log files can be accessed using the DB Console, which displays them in JSON format. 1. Access the DB Console and then click **Advanced Debug** in the left-hand navigation. 2. Under **Raw Status Endpoints (JSON)**, click **Log Files** to view the JSON of all collected logs. 3. Copy one of the log filenames. Then click **Specific Log File** and replace the `cockroach.log` placeholder in the URL with the filename. #### Known limitations Log files can only be accessed in the DB Console if they are stored in the same directory as the file sink for the `DEV` channel. ### Output to Fluentd-compatible network collectors CockroachDB can send logs over the network to a [Fluentd](https://www.fluentd.org/)-compatible log collector (e.g., [Elasticsearch](https://www.elastic.co/elastic-stack), [Splunk](https://www.splunk.com/)). `fluent-servers` specifies the channels that output to a server, along with the server configuration details. For example: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} file-defaults: ... fluent-defaults: ... http-defaults: ... sinks: fluent-servers: health: channels: [HEALTH] address: 127.0.0.1:5170 ... ``` A Fluentd sink can be listed more than once with different `address` values. This routes the same logs to different Fluentd servers. Along with the [common sink parameters](#common-sink-parameters), each Fluentd server accepts the following additional parameters: | Parameter | Description | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `channels` | List of channels that output to this sink. Use a YAML array or string of channel names, `ALL` to include all channels, or `ALL EXCEPT {channels}` to include all channels except the specified channel names. For more details on acceptable syntax, see [Logging channel selection](#logging-channel-selection). | | `address` | Network address and port of the log collector. | | `net` | Network protocol to use. Options are `tcp`, `tcp4`, `tcp6`, `udp`, `udp4`, `udp6`, or `unix`. **Default:**`tcp` **Note:** In most cases, use the default of `tcp` rather than `udp`. The UDP/IP protocol has a \~65 KB (65,507 bytes) limit per datagram. This limit is enforced by the operating system. Log lines that exceed this limit return a `message too long` error. | | `buffering` | Configures buffering of log messages for the sink, with the following sub-parameters:When `max-staleness` and `flush-trigger-size` are used together, whichever is reached first will trigger the flush. `buffering` is enabled by default for [Fluentd-compatible](#output-to-fluentd-compatible-network-collectors) log sinks. To explicitly disable log buffering, specify `buffering: NONE` instead. This setting is typically disabled for security-related logs. See [Log buffering](#log-buffering-for-network-sinks) for more details and usage. | For an example network logging configuration, see Logging use cases. ### Output to HTTP network collectors CockroachDB can send logs over the network to an HTTP server. `http-servers` specifies the channels that output to a server, along with the server configuration details. For example: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} file-defaults: ... fluent-defaults: ... http-defaults: ... sinks: http-servers: health: channels: [HEALTH] address: 127.0.0.1:5170 method: POST unsafe-tls: false timeout: 2s disable-keep-alives: false ... ``` An HTTP sink can be listed more than once with different `address` values. This routes the same logs to different HTTP servers. Along with the [common sink parameters](#common-sink-parameters), each HTTP server accepts the following additional parameters: | Parameter | Description | | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `channels` | List of channels that output to this sink. Use a YAML array or string of channel names, `ALL` to include all channels, or `ALL EXCEPT {channels}` to include all channels except the specified channel names. For more details on acceptable syntax, see [Logging channel selection](#logging-channel-selection). | | | `address` | Network address and port of the log collector. | | | `method` | HTTP method to use. Can be `GET` or `POST`. **Default:**`POST` | | | `unsafe-tls` | When `true`, bypasses TLS server validation. **Default:**`false` | | | `timeout` | Timeout before requests are abandoned. **Default:**`0` (no timeout) | | | `disable-keep-alives` | When `true`, disallows reuse of the server connection across requests. **Default:**`false` (reuses connections) | | | `compression` | Compression method for the HTTP request body. Valid values `gzip` or `none`. **Default:**`gzip` | | | `headers` | Map of key-value string pairs which will be appended to every request as custom HTTP headers. | | | | `file-based-headers` | Map of key-filepath string pairs. The file specified by the filepath only contains a value that corresponds to the key. The key from the YAML configuration and the value contained in the file will be appended to every request as a custom HTTP header. For example: `file-based-headers: {X-CRDB-HEADER-KEY: /some/path/to/file\_with\_value, X-ANOTHER-HEADER-KEY: /other/path/to/file\_with\_another\_value}` A value in a file can be updated without restarting the `cockroach` process. Instead, send SIGHUP to the `cockroach` process to notify it to refresh values. | | `buffering` | Configures buffering of log messages for the sink, with the following sub-parameters:When `max-staleness` and `flush-trigger-size` are used together, whichever is reached first will trigger the flush. `buffering` is enabled by default for [HTTP](#output-to-http-network-collectors) log sinks. To explicitly disable log buffering, specify `buffering: NONE` instead. This setting is typically disabled for security-related logs. See [Log buffering](#log-buffering-for-network-sinks) for more details and usage. | | For an example network logging configuration, see Logging use cases. For an example that uses `compression`, `headers`, `file-based-headers`, and `buffering` parameters, see Configure an HTTP network collector for Datadog. ### Output to `stderr` CockroachDB can output messages to the [standard error stream (`stderr`)](https://wikipedia.org/wiki/Standard_streams#Standard_error_\(stderr\)), which prints them to the machine's terminal but does not store them. `stderr` specifies the channels that output to the stream. For example: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} file-defaults: ... fluent-defaults: ... http-defaults: ... sinks: stderr: channels: [DEV] ``` Along with the [common sink parameters](#common-sink-parameters), `stderr` accepts the following additional parameters: The `format` parameter for `stderr` can be set to any one of the valid log formats. It is set to `crdb-v2-tty` by default. | Parameter | Description | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `channels` | List of channels that output to this sink. Use a YAML array or string of channel names, `ALL` to include all channels, or `ALL EXCEPT {channels}` to include all channels except the specified channel names. For more details on acceptable syntax, see [Logging channel selection](#logging-channel-selection). | | `no-color` | When `true`, removes terminal color [escape codes](https://wikipedia.org/wiki/ANSI_escape_code) from the output. | Because server start-up messages are always emitted at the start of the standard error stream, it is generally difficult to automate integration of `stderr` with log analyzers. We recommend using [file logging](#output-to-files) or network logging via [Fluentd](#output-to-fluentd-compatible-network-collectors) or [HTTP](#output-to-http-network-collectors) instead of `stderr` when integrating with automated monitoring software. By default, `cockroach start` and `cockroach start-single-node` do not print any messages to `stderr`. However, if the `cockroach` process does not have access to on-disk storage, all messages are printed to `stderr`. ### Logging channel selection Log channels cannot be completely disabled. You can configure how log messages are routed to different sinks and adjust verbosity levels, but you cannot turn off a log channel entirely. For example, a channel that **is not explicitly configured** for a file group **is enabled** in the `default` file group. For each sink, multiple channels can be selected. Note that: * Spacing between items will vary according to the syntax used. * Channel selection is case-insensitive. These selections are equivalent: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} # Select OPS and HEALTH. channels: [OPS, HEALTH] channels: 'OPS, HEALTH' channels: OPS,HEALTH channels: - OPS - HEALTH ``` By default, the severity level set by `filter` in the [sink configuration](#common-sink-parameters) is used. However, you can specify a different severity level for each channel. For more information on severity levels, see [Set logging levels](#set-logging-levels). These selections are equivalent: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} # Select PERF at severity INFO, and HEALTH and OPS at severity WARNING. channels: {INFO: [PERF], WARNING: [HEALTH, OPS]} channels: INFO: - PERF WARNING: - OPS - HEALTH ``` Brackets are optional when selecting a single channel: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} channels: OPS channels: {INFO: PERF} ``` To select all channels, using the `all` keyword: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} channels: all channels: 'all' channels: [all] channels: ['all'] ``` To select all channels except for a subset, using the `all except` keyword prefix: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} channels: all except ops,health channels: all except [ops,health] channels: 'all except ops, health' channels: 'all except [ops, health]' ``` ## Configure logging defaults When setting up a logging configuration, it's simplest to define shared parameters in `file-defaults` and `fluent-defaults` and override specific values as needed in [`file-groups`](#output-to-files), [`fluent-servers`](#output-to-fluentd-compatible-network-collectors), [`http-servers`](#output-to-http-network-collectors), and [`stderr`](#output-to-stderr). For a complete example, see the [default configuration](#default-logging-configuration). You can view your current settings by running `cockroach debug check-log-config`, which returns the YAML definitions and a URL to a visualization of the current logging configuration. This section describes some recommended defaults. ### Set file defaults Defaults for log files are set in `file-defaults`, which accepts all [common sink parameters](#common-sink-parameters) and the [file group parameters](#output-to-files) `dir`, `max-file-size`, `max-group-size`, `file-permissions`, and `buffered-writes`. #### Logging directory By default, CockroachDB adds log files to a `logs` subdirectory in the first on-disk `store` directory (default: `cockroach-data`): ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} cockroach-data/logs ``` Each Cockroach node generates log files in the directory specified by its logging configuration. These logs detail the internal activity of that node without visibility into the behavior of other nodes. When troubleshooting, it's best to refer to the output directory for the cluster log files, which collect the messages from all active nodes. In cloud deployments, the main data store will be subject to an IOPS budget. Adding logs to the store directory will excessively consume IOPS. For this reason, cloud deployments should output log files to a separate directory with fewer IOPS restrictions. You can override the default logging directory like this: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} file-defaults: dir: /custom/dir/path/ ``` #### File logging format The default message format for log files is `crdb-v2`. Each `crdb-v2` log message starts with a flat prefix that contains event metadata (e.g., severity, date, timestamp, channel), followed by the event payload. For details on the metadata, see Log formats. If you plan to read logs programmatically, you can switch to a JSON or compact JSON format: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} file-defaults: format: json ``` `format` refers to the envelope of the log message, which contains the event metadata. This is separate from the event payload, which corresponds to its event type. ### Set Fluentd defaults Defaults for Fluentd-compatible network sinks are set in `fluent-defaults`, which accepts all [common sink parameters](#common-sink-parameters). Note that the [server parameters](#output-to-fluentd-compatible-network-collectors) `address` and `net` are *not* specified in `fluent-defaults`: * `address` must be specified for each sink under `fluent-servers`. * `net` is not required and defaults to `tcp`. #### Fluentd logging format The default message format for network output is `json-fluent-compact`. Each log message is structured as a JSON payload that can be read programmatically. The `json-fluent-compact` and `json-fluent` formats include a `tag` field that is required by the [Fluentd protocol](https://docs.fluentd.org/configuration/config-file). For details, see Log formats. ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} fluent-defaults: format: json-fluent ``` `format` refers to the envelope of the log message. This is separate from the event payload, which is structured according to event type. ### Set HTTP defaults Defaults for HTTP sinks are set in `http-defaults`, which accepts all [common sink parameters](#common-sink-parameters). Note that the [server parameters](#output-to-http-network-collectors) `address` and `method` are *not* specified in `fluent-defaults`: * `address` must be specified for each sink under `http-servers`. * `method` is not required and defaults to `POST`. #### HTTP logging format The default message format for HTTP output is `json-compact`. Each log message is structured as a JSON payload that can be read programmatically. For details, see Log formats. ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} http-defaults: format: json ``` `format` refers to the envelope of the log message. This is separate from the event payload, which is structured according to event type. ### Set logging levels Log messages are associated with a severity level when they are generated. Each logging sink accepts messages from each logging channel at a minimum severity level. This minimum severity level can be specified [per sink](#common-sink-parameters) or by default using the `filter` attribute. Messages with severity levels below the configured threshold do not enter logging channels and are discarded. The [default configuration](#default-logging-configuration) uses the following severity levels for `cockroach start` and `cockroach start-single-node`: * `file-defaults`, `fluent-defaults`, and `http-defaults` each use `filter: INFO`. Since `INFO` is the lowest severity level, file and network sinks will emit all log messages. * `stderr` uses `filter: NONE` and does not emit log messages. * The `default` file group uses `filter: INFO` for events from the `DEV` and `OPS` channels, and `filter: WARNING` for all others. All other `cockroach` commands use `filter: WARNING` and log to `stderr` by default, with these exceptions: * `cockroach workload` uses `filter: INFO`. * `cockroach demo` uses `filter: NONE` (discards all log messages). You can override the `file-defaults`, `fluent-defaults`, and `http-defaults` severity levels on a per-sink basis. For example, this will include `DEV` events at severity `WARNING`: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} sinks: file-groups: dev: channels: DEV filter: WARNING ``` You can also override the `filter` attribute and set severity levels on a [per-channel](#logging-channel-selection) basis. For example, this will include `DEV` events at severity `INFO`, and `OPS` events at severity `ERROR`: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} sinks: file-groups: dev: channels: {INFO: DEV, ERROR: OPS} ``` ### Set timezone Using the `format-options` sink parameter, you may set the timezone for a specified log format. #### `timezone` for text formats The log output formats `crdb-v1` and `crdb-v2` support the format option `timezone`. When specified, the corresponding timezone is used to produce the timestamp column. The value can be any [timezone name recognized by the Go standard library](https://github.com/arp242/tz/blob/bf333631bec4/list.go). For example: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} file-defaults: format: crdb-v2 format-options: {timezone: america/new_york} ``` The timezone offset is included with the timestamp to ensure that the times can be read back precisely. Example logging output: ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} I231030 14:17:23.674909-040000 1 1@cli/log_flags.go:200 ⋮ [n?] 1 using explicit logging configuration: I231030 14:17:23.674909-040000 1 1@cli/log_flags.go:200 ⋮ [n?] 1 +‹file-defaults:› I231030 14:17:23.674909-040000 1 1@cli/log_flags.go:200 ⋮ [n?] 1 +‹ format: crdb-v2› I231030 14:17:23.674909-040000 1 1@cli/log_flags.go:200 ⋮ [n?] 1 +‹ format-options: {timezone: america/new_york}› ^^^^^^^ indicates GMT-4 timezone offset was used. ``` #### `datetime` field for JSON format The log output format `json` supports the format options `datetime-format` and `datetime-timezone`. `datetime-format` controls whether a `datetime` field with human-readable timestamp values is included in the logs. Valid values for this option are: * `none`: Disable the creation of the `datetime` field. This is the default value. * `iso8601` or `rfc3339`: Format the timestamp like "2006-01-02T15:04:05.999999999Z". * `rfc1123`: Format the timestamp like "Mon, 02 Jan 2006 15:04:05 +0000". The existing `timestamp` field with epoch values remains unchanged for backward-compatibility. `datetime-timezone` selects which timezone to use when formatting the `datetime` field. The value can be any [timezone name recognized by the Go standard library](https://github.com/arp242/tz/blob/bf333631bec4/list.go). The default value is `UTC`. `datetime-timezone` must be combined with `datetime-format` set to a valid format since the default value for the `datetime-format` option is `none` and therefore the `datetime` field is not included by default. For example: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} file-defaults: format: json format-options: { datetime-format: rfc3339, datetime-timezone: America/New_York } ``` Example logging output: ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} {"channel_numeric":1,"channel":"OPS","timestamp":"1699305954.983614000","datetime":"2023-11-06T16:25:54.983614-05:00","version":"v23.2.0","severity_numeric":1,"severity":"INFO","goroutine":1,"file":"cli/log_flags.go","line":200,"entry_counter":1,"redactable":1,"tags":{"n":"?"},"message":"using explicit logging configuration:\n‹file-defaults:›\n‹ format: json›\n‹ format-options: { datetime-format: rfc3339, datetime-timezone: America/New_York }›"} ``` Enabling the `datetime` field introduces CPU overhead and is nearly always unnecessary. When sending output to a log collector (for example, via [Fluent](#output-to-fluentd-compatible-network-collectors) or [Datadog](#output-to-http-network-collectors)), the log collector can be configured to transform the epoch values in the `timestamp` field into a human-readable format. When inspecting a `json` formatted log file produced by CockroachDB, you can use the command `cockroach debug merge-logs` to convert the log into `crdb-v1` format which includes timestamps in the `rfc3339` format. ### Redact logs CockroachDB can redact personally identifiable information (PII) from log messages. The logging system includes two parameters that handle this differently: * `redact` is disabled by default. When enabled, `redact` automatically redacts sensitive data from logging output. We do *not* recommend enabling this on the `DEV` channel because it impairs our ability to troubleshoot problems. * `redactable` is enabled by default. This places redaction markers around sensitive fields in log messages. These markers are recognized by `cockroach debug zip` and `cockroach debug merge-logs`, which aggregate CockroachDB log files and can be instructed to redact sensitive data from their output. When collecting logs centrally (e.g., in data mining scenarios where non-privileged users have access to logs) or over a network (e.g., to an external log collector), it's safest to enable `redact`: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} file-defaults: redact: true fluent-defaults: redact: true http-defaults: redact: true ``` In addition, the `DEV` channel should be output to a separate logging directory, since it is likely to contain sensitive data. See [`DEV` channel](#dev-channel). External log collectors can misinterpret the `cockroach debug` redaction markers (`< >`), since they are specific to CockroachDB. To prevent this issue when using network sinks, disable `redactable`: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} fluent-defaults: redactable: false ``` If the default redaction behavior and policies do not meet redaction requirements, we recommend using the external log collectors with the redaction markers (`< >`) to redact. In this case, enable `redactable`. ### DEV channel The `DEV` channel is used for debug and uncategorized messages. It can therefore be noisy and contain sensitive (PII) information. We recommend configuring `DEV` separately from the other logging channels. When sending logs to a [Fluentd-compatible](#output-to-fluentd-compatible-network-collectors) or [HTTP](#output-to-http-network-collectors) network collector, `DEV` logs should also be excluded from network collection. In this example, the `dev` file group is reserved for `DEV` logs. These are output to a `cockroach-dev.log` file in a custom disk `dir`: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} file-defaults: ... fluent-defaults: ... sinks: file-groups: dev: channels: [DEV] dir: /custom/dir/path/ ... ``` To ensure that you are protecting sensitive information, also [redact your logs](#redact-logs). ## Log buffering for network sinks Both [Fluentd-compatible](#output-to-fluentd-compatible-network-collectors) and [HTTP](#output-to-http-network-collectors) log sinks support the buffering of log messages by default. Previous to version v22.2, log buffering was only available for the [log file](#output-to-files) log sink. With log buffering configured, log messages are held in a buffer for a configurable time period or accumulated message size threshold before being written to the target log sink together as a batch. Log buffering helps to ensure consistent low-latency log message writes over the network even in high-traffic, high-contention scenarios. The following shows a basic log configuration with buffering configured for a [Fluentd-compatible](#output-to-fluentd-compatible-network-collectors) log sink: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} fluent-defaults: buffering: max-staleness: 20s flush-trigger-size: 2MiB max-buffer-size: 100MiB sinks: fluent-servers: health: channels: HEALTH buffering: max-staleness: 2s # Override max-staleness for HEALTH channel only ``` With this logging configuration: * CockroachDB will hold log messages in a buffer for up to `20s` (the `max-staleness` setting), or up to a collected size of `2MiB` (the `flush-trigger-size` setting), before writing ("flushing") the buffer to the log file. When both settings are used, whichever case is met first triggers the buffer flush. * If at any point the accumulated message size of buffer flushes (as triggered by reaching either the configured `max-staleness` or `flush-trigger-size` value) exceeds `100MiB` (the `max-buffer-size` setting), all new incoming log messages received are dropped until the accumulated message size in the buffer once more falls below this value. * For the `HEALTH` log channel only, override the `file-defaults` value of `20s` for `max-staleness`, instead flushing messages to the log file within up to `2s`. Logs are flushed automatically upon CockroachDB startup only if CockroachDB is managed using `systemd`. Alternatively, you may explicitly disable log buffering by setting `buffering` to `NONE`. The following log configuration explicitly disables log buffering for just the `OPS` channel on a [Fluentd-compatible](#output-to-fluentd-compatible-network-collectors) log sink: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} file-defaults: ... fluent-defaults: ... sinks: fluent-servers: ops: channels: OPS buffering: NONE ``` The following disables log buffering completely for the [Fluentd-compatible](#output-to-fluentd-compatible-network-collectors) sink: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} fluent-defaults: buffering: NONE ``` ## Stray error capture Certain events, such as uncaught software exceptions (panics), bypass the CockroachDB logging system. However, they can be useful in troubleshooting. For example, if CockroachDB crashes, it normally logs a stack trace to what caused the problem. To ensure that these stray errors can be tracked, CockroachDB does not send them to `stderr` by default. Instead, stray errors are output to a `cockroach-stderr.log` file in the default [logging directory](#logging-directory). You can change these settings in `capture-stray-errors`: ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} file-defaults: ... fluent-defaults: ... sinks: ... capture-stray-errors: enable: true dir: /custom/dir/path/ ``` When `capture-stray-errors` is disabled, [`redactable`](#redact-logs) cannot be enabled on the `stderr` sink. This is because `stderr` will contain both stray errors and logged events and cannot apply redaction markers in a reliable way. Note that [`redact`](#redact-logs) can still be enabled on `stderr` in this case. ## Default logging configuration The YAML payload below represents the default logging behavior of `cockroach start` and `cockroach start-single-node`. ```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}} file-defaults: max-file-size: 10MiB max-group-size: 100MiB file-permissions: 644 buffered-writes: true filter: INFO format: crdb-v2 redact: false redactable: true exit-on-error: true auditable: false fluent-defaults: filter: INFO format: json-fluent-compact redact: false redactable: true exit-on-error: false auditable: false http-defaults: method: POST unsafe-tls: false timeout: 0s disable-keep-alives: false filter: INFO format: json-compact redact: false redactable: true exit-on-error: false auditable: false sinks: file-groups: default: channels: INFO: [DEV, OPS] WARNING: all except [DEV, OPS] health: channels: [HEALTH] kv-distribution: channels: [KV_DISTRIBUTION] pebble: channels: [STORAGE] security: channels: [PRIVILEGES, USER_ADMIN] auditable: true sql-audit: channels: [SENSITIVE_ACCESS] auditable: true sql-auth: channels: [SESSIONS] auditable: true sql-exec: channels: [SQL_EXEC] sql-slow: channels: [SQL_PERF] sql-slow-internal-only: channels: [SQL_INTERNAL_PERF] telemetry: channels: [TELEMETRY] max-file-size: 100KiB max-group-size: 1.0MiB stderr: channels: all filter: NONE redact: false redactable: true exit-on-error: true capture-stray-errors: enable: true max-group-size: 100MiB ``` For high-traffic deployments that [output log messages to files](#output-to-files), consider raising `file-defaults: {max-group-size}` to `500MiB` or `1GiB` to extend log retention. Note that a default `dir` is not specified for `file-defaults` and `capture-stray-errors`: * The default `dir` for `file-defaults` is inferred from the first on-disk `store` directory. See [Logging directory](#logging-directory). * The default `dir` for `capture-stray-errors` is inherited form `file-defaults`. ## See also * Logging Use Cases * Log Formats