> ## Documentation Index
> Fetch the complete documentation index at: https://cockroachlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrade to CockroachDB v23.1

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

Because of CockroachDB's <InternalLink path="multi-active-availability">multi-active availability</InternalLink> design, you can perform a "rolling upgrade" of your CockroachDB cluster. This means that you can upgrade nodes one at a time without interrupting the cluster's overall health and operations.

This page describes how to upgrade to the latest **v23.1** release, (<InternalLink version="releases" path="release-support-policy#support-types">LTS</InternalLink>). To upgrade CockroachDB on Kubernetes, refer to <InternalLink path="upgrade-cockroachdb-kubernetes">single-cluster</InternalLink> or <InternalLink path="orchestrate-cockroachdb-with-kubernetes-multi-cluster#upgrade-the-cluster">multi-cluster</InternalLink> instead.

## Terminology

Before upgrading, review the CockroachDB [release](../releases/) terminology:

* A new *major release* is performed multiple times per year. The major version number indicates the year of release followed by the release number, starting with 1. For example, the latest major release is .
* Each <InternalLink version="releases" path="release-support-policy">supported</InternalLink> major release is maintained across *patch releases* that contain improvements including performance or security enhancements and bug fixes. Each patch release increments the major version number with its corresponding patch number. For example, patch releases of  use the format .x.
* All major and patch releases are suitable for production environments, and are therefore considered "production releases". For example, the latest production release is .
* Prior to an upcoming major release, alpha, beta, and release candidate (RC) binaries are made available for users who need early access to a feature before it is available in a production release. These releases append the terms `alpha`, `beta`, or `rc` to the version number. These "testing releases" are not suitable for production environments and are not eligible for support or uptime SLA commitments. For more information, refer to the <InternalLink version="releases" path="release-support-policy">Release Support Policy</InternalLink>.

<Note>
  There are no "minor releases" of CockroachDB.
</Note>

## Step 1. Verify that you can upgrade

<Danger>
  In CockroachDB v22.2.x and above, a cluster that is upgraded to an alpha binary of CockroachDB or a binary that was manually built from the `master` branch cannot subsequently be upgraded to a production release.
</Danger>

Run <InternalLink path="cockroach-sql">`cockroach sql`</InternalLink> against any node in the cluster to open the SQL shell. Then check your current cluster version:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW CLUSTER SETTING version;
```

To upgrade to , you must be running either:

* **Any earlier v23.1 release:**  to .

* **A  production release:** .0 to .

If you are running any other version, take the following steps **before** continuing on this page:

| Version                   | Action(s) before upgrading to any v23.1 release                                                                                       |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| Pre-v23.1 testing release | Upgrade to a corresponding production release; then upgrade through each subsequent major release, ending with a  production release. |
| Pre- production release   | Upgrade through each subsequent major release, ending with a  production release.                                                     |
| testing release           | Upgrade to a  production release.                                                                                                     |

When you are ready to upgrade to , continue to [step 2](#step-2-prepare-to-upgrade).

## Step 2. Prepare to upgrade

Before starting the upgrade, complete the following steps.

### Ensure you have a valid license key

To perform major version upgrades, you must have <InternalLink path="licensing-faqs#obtain-a-license">a valid license key</InternalLink>.

Patch version upgrades can be performed without a valid license key, with the following limitations:

* The cluster will run without limitations for a specified grace period. During that time, alerts are displayed that the cluster needs a valid license key. For more information, refer to the <InternalLink path="licensing-faqs">Licensing FAQs</InternalLink>.
* The cluster is <InternalLink path="licensing-faqs">throttled</InternalLink> at the end of the grace period if no <InternalLink path="licensing-faqs#set-a-license">valid license key is added to the cluster</InternalLink> before then.

If you have an **Enterprise Free** or **Enterprise Trial** license, you must enable telemetry using the <InternalLink path="cluster-settings">`diagnostics.reporting.enabled`</InternalLink> cluster setting, as shown below in order to finalize a major version upgrade:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING diagnostics.reporting.enabled = true;
```

If a cluster with an **Enterprise Free** or **Enterprise Trial** license is upgraded across patch versions and does not meet telemetry requirements:

* The cluster will run without limitations for a 7-day grace period. During that time, alerts are displayed that the cluster needs to send telemetry.
* The cluster is <InternalLink path="licensing-faqs">throttled</InternalLink> if telemetry is not received before the end of the grace period.

For more information, refer to the <InternalLink path="licensing-faqs">Licensing FAQs</InternalLink>.

If you want to stay on the previous version, you can roll back the upgrade before finalization.

### Review breaking changes

Review the <InternalLink version="releases" path="v23.1">backward-incompatible changes</InternalLink>, <InternalLink version="releases" path="v23.1">deprecated features</InternalLink>, and <InternalLink version="releases" path="v23.1">key cluster setting changes</InternalLink> in v23.1. If any affect your deployment, make the necessary changes before starting the rolling upgrade to v23.1.

### Check load balancing

Make sure your cluster is behind a <InternalLink path="recommended-production-settings#load-balancing">load balancer</InternalLink>, or your clients are configured to talk to multiple nodes. If your application communicates with a single node, stopping that node to upgrade its CockroachDB binary will cause your application to fail.

### Check cluster health

Verify the overall health of your cluster using the <InternalLink path="ui-cluster-overview-page">DB Console</InternalLink>:

* Under **Node Status**, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as `SUSPECT` or `DEAD`, identify why the nodes are offline and either restart them or [decommission](node-shutdown.html?filters=decommission#remove-nodes) them before beginning your upgrade. If there are `DEAD` and non-decommissioned nodes in your cluster, it will not be possible to finalize the upgrade (either automatically or manually).

* Under **Replication Status**, make sure there are `0` under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it's important to identify and resolve the cause of range under-replication and/or unavailability before beginning your upgrade.

* In the **Node List**:
  * Make sure all nodes are on the same version. If any nodes are behind, upgrade them to the cluster's current version first, and then start this process over.

* In the **Metrics** dashboards:
  * Make sure <InternalLink path="common-issues-to-monitor#cpu-usage">CPU</InternalLink>, <InternalLink path="common-issues-to-monitor#database-memory-usage">memory</InternalLink>, and <InternalLink path="common-issues-to-monitor#storage-capacity">storage</InternalLink> capacity are within acceptable values for each node. Nodes must be able to tolerate some increase in case the new version uses more resources for your workload. If any of these metrics is above healthy limits, consider <InternalLink path="cockroach-start">adding nodes</InternalLink> to your cluster before beginning your upgrade.

### Check decommissioned nodes

If your cluster contains partially-decommissioned nodes, they will block an upgrade attempt.

1. To check the status of decommissioned nodes, run  the <InternalLink path="cockroach-node#show-the-status-of-all-nodes">`cockroach node status --decommission`</InternalLink> command:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   cockroach node status --decommission
   ```

   In the output, verify that the value of the `membership` field of each node is `decommissioned`. If any node's `membership` value is `decommissioning`, that node is not fully decommissioned.

2. If any node is not fully decommissioned, try the following:

   1. First, reissue the [decommission command](node-shutdown.html?filters=decommission#decommission-the-node). The second command typically succeeds within a few minutes.
   2. If the second decommission command does not succeed, [recommission](node-shutdown.html?filters=decommission#recommission-nodes) and then decommission it again. Before continuing the upgrade, the node must be marked as `decommissioned`.

### Back up cluster

Because CockroachDB is designed with high fault tolerance, backups are primarily needed for <InternalLink path="disaster-recovery">disaster recovery</InternalLink>. However, taking regular backups of your data is an operational best practice. When upgrading to a major release, **we recommend <InternalLink path="backup-and-restore-overview">taking a backup</InternalLink> of your cluster**.
See our <InternalLink path="restoring-backups-across-versions#support-for-restoring-backups-into-a-newer-version">support policy for restoring backups across versions</InternalLink>.

### Pause changefeed jobs

<InternalLink path="pause-job">Pause</InternalLink> running <InternalLink path="change-data-capture-overview">changefeed</InternalLink> jobs before starting the rolling upgrade process:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
PAUSE JOB {changefeed_job_ID};
```

Or, pause all changefeed jobs:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
PAUSE ALL CHANGEFEED JOBS;
```

<Danger>
  During a rolling upgrade, running changefeed jobs can slow down as more of the nodes move to the later version of CockroachDB. We recommend pausing changefeed jobs before upgrading and <InternalLink path="resume-job">resuming</InternalLink> changefeeds once the upgrade is finalized.
</Danger>

For more details on the potential impacts of node restarts, refer to the <InternalLink path="changefeed-messages#node-restarts">Changefeed Messages</InternalLink> page.

### Reset SQL statistics

Before upgrading to CockroachDB v23.1, it is recommended to reset the cluster's <InternalLink path="crdb-internal">SQL statistics</InternalLink>. Otherwise, it may take longer for the upgrade to complete on a cluster with large statement or transaction statistics tables. This is due to the addition of a new column and a new index to these tables. To reset SQL statistics, issue the following SQL command:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SELECT crdb_internal.reset_sql_stats();
```

## Step 3. Decide how the upgrade will be finalized

This step is relevant only when upgrading from .x to v23.1. For upgrades within the v23.1.x series, skip this step.

By default, after all nodes are running the new version, the upgrade process will be **auto-finalized**. This will enable certain [features and performance improvements introduced in v23.1](#features-that-require-upgrade-finalization). However, it will no longer be possible to [roll back to ](#step-5-roll-back-the-upgrade-optional) if auto-finalization is enabled. In the event of a catastrophic failure or corruption, the only option will be to start a new cluster using the previous binary and then restore from one of the backups created prior to performing the upgrade. For this reason, **we recommend disabling auto-finalization** so you can monitor the stability and performance of the upgraded cluster before finalizing the upgrade, but note that you will need to follow all of the subsequent directions, including the manual finalization in [step 6](#step-6-finish-the-upgrade):

1. Upgrade to , if you haven't already.

2. Start the <InternalLink path="cockroach-sql">`cockroach sql`</InternalLink> shell against any node in the cluster.

3. Set the `cluster.preserve_downgrade_option` <InternalLink path="cluster-settings">cluster setting</InternalLink>:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   SET CLUSTER SETTING cluster.preserve_downgrade_option = '';
   ```

   It is only possible to set this setting to the current cluster version.

### Features that require upgrade finalization

When upgrading from  to v23.1, certain features and performance improvements will be enabled only after finalizing the upgrade, including but not limited to:

* The <InternalLink path="create-schedule-for-changefeed">`CREATE SCHEDULE FOR CHANGEFEED`</InternalLink> statement allows you to create scheduled changefeeds.
* The <InternalLink path="security-reference/authorization#supported-privileges">`MODIFYSQLCLUSTERSETTING` and `VIEWJOB` system privileges</InternalLink>.

For an expanded list of features included in the v23.1 release, see the <InternalLink version="releases" path="v23.1">v23.1 release notes</InternalLink>.

## Step 4. Perform the rolling upgrade

<Tip>
  Cockroach Labs recommends creating scripts to perform these steps instead of performing them manually.
</Tip>

Follow these steps to perform the rolling upgrade. To upgrade CockroachDB on Kubernetes, refer to <InternalLink path="upgrade-cockroachdb-kubernetes">single-cluster</InternalLink> or <InternalLink path="orchestrate-cockroachdb-with-kubernetes-multi-cluster#upgrade-the-cluster">multi-cluster</InternalLink> instead.

For each node in your cluster, complete the following steps. Be sure to upgrade only one node at a time, and wait at least one minute after a node rejoins the cluster to upgrade the next node. Simultaneously upgrading more than one node increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability.

<Danger>
  After beginning a major-version upgrade, Cockroach Labs recommends upgrading all nodes as quickly as possible. In a cluster with nodes running different major versions of CockroachDB, a query that is sent to an upgraded node can be distributed only among other upgraded nodes. Data accesses that would otherwise be local may become remote, and the performance of these queries can suffer.
</Danger>

These steps perform an upgrade to the latest v23.1 release.

1. <InternalLink path="node-shutdown#perform-node-shutdown">Drain and shut down the node</InternalLink>.

2. Visit <InternalLink version="releases" path="v23.1">What's New in v23.1?</InternalLink> and download the **CockroachDB  full binary** for your architecture.

3. Extract the archive. In the following instructions, replace <code>{'{COCKROACHDB_DIR}'}</code> with the path to the extracted archive directory.

4. If you have a previous version of the `cockroach` binary in your `$PATH`, rename the outdated `cockroach` binary, and then move the new one into its place.

   If you get a permission error because the `cockroach` binary is located in a system directory, add `sudo` before each command. The binary will be owned by the effective user, which is `root` if you use `sudo`.

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   i="$(which cockroach)"; mv "$i" "$i"_old
   ```

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   cp -i {COCKROACHDB_DIR}/cockroach /usr/local/bin/cockroach
   ```

5. If a cluster has corrupt descriptors, a major-version upgrade cannot be finalized. In CockroachDB v23.1.6 and subsequent 23.1 versions, automatic descriptor repair is enabled and cannot be disabled. In CockroachDB v23.1.11 and above, automatic descriptor repair is available but disabled by default. To enable it in these versions, set the environment variable `COCKROACH_RUN_FIRST_UPGRADE_PRECONDITION` to `true` after installing the v23.1 binary but before restarting the `cockroach` process on the node. Monitor the <InternalLink path="logging">cluster logs</InternalLink> for errors. If a descriptor cannot be repaired automatically, [contact support](https://support.cockroachlabs.com/hc) for assistance completing the upgrade.

6. Start the node so that it can rejoin the cluster.

   Without a process manager like `systemd`, re-run the <InternalLink path="cockroach-start">`cockroach start`</InternalLink> command that you used to start the node initially, for example:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   cockroach start \
       --certs-dir=certs \
       --advertise-addr={node address} \
       --join={node1 address},{node2 address},{node3 address}
   ```

   If you are using `systemd` as the process manager, run this command to start the node:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   systemctl start {systemd config filename}
   ```

   Re-run the <InternalLink path="cockroach-start">`cockroach start`</InternalLink> command that you used to start the node initially, for example:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   cockroach start \
       --certs-dir=certs \
       --advertise-addr={node address} \
       --join={node1 address},{node2 address},{node3 address}
   ```

7. Verify the node has rejoined the cluster through its output to <InternalLink path="cockroach-start#standard-output">`stdout`</InternalLink> or through the <InternalLink path="ui-cluster-overview-page#node-status">DB Console</InternalLink>.

8. If you use `cockroach` in your `$PATH`, you can remove the previous binary:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   rm /usr/local/bin/cockroach_old
   ```

   If you leave versioned binaries on your servers, you do not need to do anything.

9. After the node has rejoined the cluster, ensure that the node is ready to accept a SQL connection.

   Unless there are tens of thousands of ranges on the node, it's usually sufficient to wait one minute. To be certain that the node is ready, run the following command:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   cockroach sql -e 'select 1'
   ```

   The command will automatically wait to complete until the node is ready.

10. Repeat these steps for the next node.

## Step 5. Roll back the upgrade (optional)

If you decide to roll back to , you must do so before the upgrade has been [finalized](#step-6-finish-the-upgrade), as described in the next section. It is always possible to roll back to a previous v23.1 version.

To roll back an upgrade, do the following on each cluster node:

1. [Perform a rolling upgrade](#step-4-perform-the-rolling-upgrade), as described in the previous section, but replace the upgraded `cockroach` binary on each node with the binary for the previous version.
2. Restart the `cockroach` process on the node and verify that it has rejoined the cluster before rolling back the upgrade on the next node.
3. After all nodes have been rolled back and rejoined the cluster, [finalize the rollback](#step-6-finish-the-upgrade) in the same way as you would finalize an upgrade, as described in the next section.

## Step 6. Finish the upgrade

Because a finalized major-version upgrade cannot be rolled back, Cockroach Labs recommends that you monitor the stability and performance of your cluster with the upgraded binary for at least a day before deciding to finalize the upgrade.

Finalization is required only when upgrading from .x to v23.1. For upgrades within the v23.1.x series, skip this step.

1. If you disabled auto-finalization in [step 3](#step-3-decide-how-the-upgrade-will-be-finalized), monitor the stability and performance of your cluster for at least a day. If you decide to roll back the upgrade, repeat the [rolling restart procedure](#step-4-perform-the-rolling-upgrade) with the previous binary. Otherwise, perform the following steps to re-enable upgrade finalization to complete the upgrade to v23.1. Cockroach Labs recommends that you either finalize or roll back a major-version upgrade within a relative short period of time; running in a partially-upgraded state is not recommended.

   A cluster that is not finalized on  cannot be upgraded to v23.1 until the  upgrade is finalized.

2. Once you are satisfied with the new version, run <InternalLink path="cockroach-sql">`cockroach sql`</InternalLink> against any node in the cluster to open the SQL shell.

3. Re-enable auto-finalization:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > RESET CLUSTER SETTING cluster.preserve_downgrade_option;
   ```

   A series of migration jobs runs to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. Until the upgrade is finalized, these features and functions will not be available and the command `SHOW CLUSTER SETTING version` will return \`\`.

   You can monitor the process of the migration in the DB Console <InternalLink path="ui-jobs-page">Jobs page</InternalLink>. Migration jobs have names in the format `-{migration-id}`. If a migration job fails or stalls, Cockroach Labs can use the migration ID to help diagnose and troubleshoot the problem. Each major version has different migration jobs with different IDs.

<Note>
  All <InternalLink path="online-schema-changes">schema change</InternalLink> jobs must reach a terminal state before finalization can complete. Finalization can therefore take as long as the longest-running schema change. Otherwise, the amount of time required for finalization depends on the amount of data in the cluster, as the process runs various internal maintenance and migration tasks. During this time, the cluster will experience a small amount of additional load.
</Note>

When all migration jobs have completed, the upgrade is complete.

1. To confirm that finalization has completed, check the cluster version:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > SHOW CLUSTER SETTING version;
   ```

   If the cluster continues to report that it is on the previous version, finalization has not completed. If auto-finalization is enabled but finalization has not completed, check for the existence of <InternalLink path="node-shutdown?filters=decommission#status-change">decommissioning nodes</InternalLink> where decommission has stalled. In most cases, issuing the `decommission` command again resolves the issue. If you have trouble upgrading, [contact Support](https://cockroachlabs.com/support/hc/).

2. If you paused <InternalLink path="change-data-capture-overview">changefeed</InternalLink> jobs before starting the upgrade process, you can now <InternalLink path="resume-job">resume</InternalLink> the jobs:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   RESUME JOB {changefeed_job_ID};
   ```

   Or, resume all changefeed jobs:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   RESUME ALL CHANGEFEED JOBS;
   ```

   Check that changefeeds are running:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   SHOW CHANGEFEED JOBS;
   ```

After the upgrade to v23.1 is finalized, you may notice an increase in <InternalLink path="architecture/storage-layer#compaction">compaction</InternalLink> activity due to a background migration job within the storage engine. To observe the migration's progress, check the **Compactions** section of the <InternalLink path="ui-storage-dashboard">Storage Dashboard</InternalLink> in the DB Console or monitor the `storage.marked-for-compaction-files` <InternalLink path="metrics">time-series metric</InternalLink>. When the metric's value nears or reaches `0`, the migration is complete and compaction activity will return to normal levels.

## Troubleshooting

After the upgrade has finalized (whether manually or automatically), it is no longer possible to downgrade to the previous release. If you are experiencing problems, we therefore recommend that you run the <InternalLink path="cockroach-debug-zip">`cockroach debug zip`</InternalLink> command on any cluster node to capture your cluster's state, then <InternalLink path="support-resources">open a support request</InternalLink> and share your debug zip.

In the event of catastrophic failure or corruption, it may be necessary to <InternalLink path="restore">restore</InternalLink> from a backup to a new cluster running .

## See also

* <InternalLink version="releases" path="release-support-policy">Release Support Policy</InternalLink>
* <InternalLink path="cockroach-node">View Node Details</InternalLink>
* <InternalLink path="cockroach-debug-zip">Collect Debug Information</InternalLink>
* <InternalLink path="cockroach-version">View Version Details</InternalLink>
* <InternalLink version="releases" path="v23.1">Release notes for our latest version</InternalLink>