cockroach debug zip connects to your cluster and gathers information from each active node into a single .zip file (inactive nodes are not included). For details on the .zip contents, see Files.
You can use the command in conjunction with cockroach debug zip to merge the collected logs into one file, making them easier to parse.
The files produced by
cockroach debug zip can contain highly , such as usernames, hashed passwords, and possibly table data. Use the --redact flag to configure CockroachDB to redact sensitive data when generating the .zip file (excluding range keys) if intending to share it with Cockroach Labs.Details
Use cases
cockroach debug zip is an expensive operation and impacts cluster performance.Only use this command as an emergency measure under the guidance of Cockroach Labs.Particularly fetching stack traces for all goroutines is a “stop-the-world” operation, which can momentarily but significantly increase SQL service latency. Exclude these goroutine stacks by using the --include-goroutine-stacks=false flag.debug zip is useful:
- If you experience severe or difficult-to-reproduce issues with your cluster, Cockroach Labs might ask you to send us your cluster’s debugging information using
cockroach debug zip. We recommend reducing the*.zipfile size by only retrieving debugging information for the relevant time range of the issue by using the--files-from, and/or--files-untilflags. - To collect all of your nodes’ logs, which you can then parse to locate issues. You can optionally use the flags to retrieve only the log files. For more information about logs, see . Also note:
- Nodes that are currently cannot deliver their logs over the network. For these nodes, you must log on to the machine where the
cockroachprocess would otherwise be running, and gather the files manually. - Nodes that are currently up but disconnected from other nodes (e.g., because of a ) may not be able to respond to
debug ziprequests forwarded by other nodes, but can still respond to requests for data when asked directly. In such situations, we recommend using the--hostflag to pointdebug zipat each of the disconnected nodes until data has been gathered for the entire cluster.
- Nodes that are currently cannot deliver their logs over the network. For these nodes, you must log on to the machine where the
Files
cockroach debug zip collects log files, heap profiles, CPU profiles, and goroutine dumps from the last 48 hours, by default.
The following files collected by cockroach debug zip, which are found in the individual node directories, can be filtered using the --exclude-files, --include-files, --files-from, and/or --files-until flags:
| Information | Filename |
|---|---|
cockroach-{log-file-group}.{host}.{user}.{start timestamp in UTC}.{process ID}.log | |
| Goroutine dumps | goroutine\_dump.{date-and-time}.{metadata}.double\_since\_last\_dump.{metadata}.txt.gz |
| Heap profiles | memprof.{date-and-time}.{heapsize}.pprof |
| Memory statistics | memstats.{date-and-time}.{heapsize}.txt |
| CPU profiles | cpuprof.{date-and-time} |
activequeryprof.{date-and-time}.csv |
.zip file, and cannot be filtered:
- System tables. The following system tables are not included:
system.userssystem.web_sessionssystem.join_tokenssystem.commentssystem.uisystem.zonessystem.statement_bundle_chunkssystem.statement_statisticssystem.transaction_statistics
- Cluster events
- Database details
- Schema change events
- Database, table, node, and range lists
- Node details
- Node liveness
- Gossip data
- Stack traces
- Range details
- Jobs
- CPU profiles
- A script (
hot-ranges.sh) that summarizes the hottest ranges (ranges receiving a high number of reads or writes)
Subcommands
While thecockroach debug command has a few subcommands, users are expected to use only the , , , , , and subcommands.
We recommend using the and subcommands only when directed by the .
The other debug subcommands are useful only to Cockroach Labs. Output of debug commands may contain sensitive or secret information.
Synopsis
The following flags must apply to an active CockroachDB node. If no nodes are live, you must .
Flags
Thedebug zip subcommand supports the following general-use, client connection, and logging flags.
General
| Flag | Description | |
|---|---|---|
--cpu-profile-duration | Fetch CPU profiles from the cluster with the specified sample duration in seconds. The debug zip command will block for the duration specified. A value of 0 disables this feature. Default:5s | |
--concurrency | The maximum number of nodes to concurrently poll for data. This can be any value between 1 and 15. | |
--exclude-files | Files to exclude from the generated .zip. This can be used to limit the size of the generated .zip, and affects logs, heap profiles, goroutine dumps, and/or CPU profiles. The files are specified as a comma-separated list of glob patterns. For example: --exclude-files=\*.log Note that this flag is applied after--include\_files. Use with this flag to see a list of files that will be contained in the .zip. | |
--exclude-nodes | Specify nodes to exclude from inspection as a comma-separated list or range of node IDs. For example: --exclude-nodes=1,10,13-15 | |
--files-from | Start timestamp for log file, goroutine dump, and heap profile collection. This can be used to limit the size of the generated .zip, which is increased by these files. The timestamp uses the format YYYY-MM-DD, followed optionally by HH:MM:SS or HH:MM. For example: --files-from='2021-07-01 15:00' When specifying a narrow time window, we recommend adding extra seconds/minutes to account for uncertainties such as clock drift. Default: 48 hours before now | |
--files-until | End timestamp for log file, goroutine dump, and heap profile collection. This can be used to limit the size of the generated .zip, which is increased by these files. The timestamp uses the format YYYY-MM-DD, followed optionally by HH:MM:SS or HH:MM. For example: --files-until='2021-07-01 16:00' When specifying a narrow time window, we recommend adding extra seconds/minutes to account for uncertainties such as clock drift. Default: 24 hours beyond now (to include files created during .zip creation) | |
--include-files | Files to include in the generated .zip. This can be used to limit the size of the generated .zip, and affects logs, heap profiles, goroutine dumps, and/or CPU profiles. The files are specified as a comma-separated list of glob patterns. For example: --include-files=\*.pprof Note that this flag is applied before--exclude-files. Use with this flag to see a list of files that will be contained in the .zip. | |
--include-goroutine-stacks | Fetch stack traces for all goroutines running on each targeted node in nodes/\*/stacks.txt and nodes/\*/stacks\_with\_labels.txt files. Note that fetching stack traces for all goroutines is a “stop-the-world” operation, which can momentarily have negative impacts on SQL service latency. Exclude these goroutine stacks by using the --include-goroutine-stacks=false flag. Note that any periodic goroutine dumps previously taken on the node will still be included in nodes/\*/goroutines/\*.txt.gz, as these would have already been generated and don’t require any additional stop-the-world operations to be collected. Default: true | |
--include-range-info | Include one file per node with information about the KV ranges stored on that node, in nodes/{node ID}/ranges.json. This information can be vital when debugging issues that involve the (which includes everything below the SQL layer), such as data placement, load balancing, performance or other behaviors. In certain situations, on large clusters with large numbers of ranges, these files can be omitted if and only if the issue being investigated is already known to be in another layer of the system (for example, an error message about an unsupported feature or incompatible value in a SQL schema change or statement). However, many higher-level issues are ultimately related to the underlying KV layer described by these files. Only set this to false if directed to do so by Cockroach Labs support. In addition, include problem ranges information in reports/problemranges.json. Default: true | |
--include-running-job-traces | Include information about each traceable job that is running or reverting (such as , , , ) in jobs/\*/\*/trace.zip files. This involves collecting cluster-wide traces for each running job in the cluster. Default: true | |
--nodes | Specify nodes to inspect as a comma-separated list or range of node IDs. For example: --nodes=1,10,13-15 | |
--redact | Redact sensitive data from the generated .zip, with the exception of range keys, which must remain unredacted because they are essential to support CockroachDB. This flag replaces the deprecated --redact-logs flag, which only applied to log messages contained within .zip. See Redact sensitive information for an example. | |
--redact-logs | Deprecated Redact sensitive data from collected log files only. Use the --redact flag instead, which redacts sensitive data across the entire generated .zip as well as the collected log files. Passing the --redact-logs flag will be interpreted as the --redact flag. | |
--timeout | In the process of generating a debug zip, many internal requests are made. Each request is allowed the maximum duration specified by the timeout. If an internal request does not complete within the timeout duration, an error is displayed for that request and its artifact is not included in the zip file. The timeout is suffixed with s (seconds), m (minutes), or h (hours). Default:60s |
Client connection
Logging
By default, this command logs messages tostderr. This includes events with WARNING and higher.
If you need to troubleshoot this command’s behavior, you can .
Examples
Generate a debug zip file
Generate the debug zip file for an insecure cluster:Secure examples assume you have the appropriate certificates in the default certificate directory,
${HOME}/.cockroach-certs/.